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better decision-making so that more efficient use can be made of CBI re- 
Mmmrees, both now and in the future. Methods to facilitate Actual trait 1m eo 
with CBT are discussed. These initiatives are applied specifically to the 
Navy Maintenance Training Improvement Program (MTIP). 









O DISTRIBUTION: AVAILABILITY OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION 

() UNCLASSIFIED UNLIMITED =) SAME as RFT fel pric USERS irene sit hed 
2a NAME Or RESPONSIBLE INDIVIDUAL 22b TELEPHONE (include Area Code) | «2c OFFICE SYMBOL 
fon B. Isett (408) 646-2464 Code o415 
'D FORM 1473, 84 Ma4R 83 APR edit:on may be used unt exhausted 


All other edit oe ___ SECURITY CLASSIFICATION OF THIS PAGE 
f 
: i ae U.S Government Printing Office. 1986—-606-246, 


alt imenetecetacd 


Approved for public release, distribution unlimited 


Computer-Based Training Applied to the 
U. S. Navy Maintenance Training Improvement Program 


by 


Galen Jay Ledeboer 
Lieutenant Commander, United States Navy 
A.B. ,e@omdt, College, 1973 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCTEN@E IN INFORMATION SYSTEMS 


from the 


NAVAL POSTGRADUATE SCHOOL 
september 1988 


Bor RAC. 


In the military, there has been much interest in 
combining instructional methodologies with computer 
technology to facilitate training. The potential for life 
cycle cost reduction while improving the effectiveness of 
training is very attractive. Thus, Computer-Based Training 
(CBT) has been embraced by the U. S. Navy. Initially, CBT 
was used primarily to manage training programs and has more 
recently also been used to provide instruction directly. 
However, CBT has not adequately kept pace with the 
Formidable record-keeping and data requirements of a typical 
Navy training program. Proposals are introduced here to 
improve data processing and management for CBT. Also, 
concepts are described to promote better decision-making so 
that more efficient use can be made of CBT resources, both 
now and in the future. Methods to facilitate actual 
training with CBT are discussed. These initiatives are 
applied specifically to the Navy Maintenance Training 


Improvement Program (MTIP). 
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ieee Nl RODUCT TON 


A. COMPUTER-BASED TRAINING 

In recent decades, there has been significant interest 
mrsineg individualized instruction and computer technology 
Mmemmdciiitate training. the potential cost reduction and 
learning efficiency advantages seemed readily apparent. 
Hence, much research and development has been directed to 
that area (see Appendix C). <A widely used methodology for 
momaucting training has evolved. The term to be used here 
Monwtiiat activity is Computer-Based Training (CBT). CBT is 
comprised of two major subsystems, Computer-Managed 
crotm@ction (CMI) and Computer-Aided Instruction (CAI). 

PO" Neil 8l:p. 1] CAI is also frequently referred to as 
Computer-Aided Learning (CAL). CBT, or Computer-Based 
mtiscenwertion (CBI) as it is frequently known, is primarily 
comprised of and loosely defined as the sum of these two 
components. 

CBT, CMI and CAI are selected from a proliferation of 
terminology which has developed over the years. Appendix A 
provides definitions of most associated terms found in 
emrrent literature. Appendix B lists many CBT and U. &. 
Navy-related acronyms. 

CBT development typically begins with the CMI component. 


CMI is defined as a subsystem of CBT which uses the computer 


to manage overall instructional activity... Piro anclides 
planning, administrat tony snontLonine ss tidenteynoomess:. 
allocating instructional resources and providing reports. 
CMI is the function which regulates the level of control and 
efficiency in the training system. Figure yl. Meri iictraece 
some basic components of a typical training center as they 
are related to a basic CMI subsystem. Notice that CMI 
accommodates both non-computerized and computerized 

ins br wc Ciom. 

CAI, the other major subsystem of CBT, is defined as a 
method of learning where a computer is the primary storage 
and delivery device for t2nstructivon, Fike sa eutor 
student interacts directly with a computer in an 
individualized training environment. The student works in 
real time and may receive direct feedback from the 
computer. [NPRDC TN 82-17, 82 }p@ee2 3b Kemer: p oars, | 

CMI and CAL are not entirely distiner andes joint 
components as illustrated in Figure 1.2. The primary areas 
of commonalty are control functions and the hardware used. 
CMI provides overall external controletior CAI) Howevene 
internally CAI provides control for its own processes injee 
form of an operating system and externally must interface 
with the CMI system. 

Material presented by CMI is generally not instructional 
but rather related to administrative tasks and evaluation 


processes, such as testing or collecting statistical 


Fleet 
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Mist ruction 


Support 





mecine leila jon Compomemts of a Training Center 
(bakers lew. 26] 

information for a database or a report. CMI is not 
necessarily conducted in real time and students and 
instructor generally share a few workstations. CMI is 
inherently of more benefit to the supervisor of training or 
moemristructors. Ihis 1s because it simplifies many of the 
moltlle administrative facets of their jobs, freeing them to 


spend mone instructional time with trainees. 


CMI CAI 


Administration Peedibac 
Communal 
Monitor Progress functions Individualized 


Planning Hardware Interactive 


Reports Tutorial 





CBT = Computer-Based Training 


Chit 


Computer-—-Managed Instruction 
CAl = Computer -fided eine rriction 


Fiegure 1.2 Components of Computer-Based Training 
[Deano - peels 2 


CAI is instructional in nature and can provide intensive 
training. More workstations are required per student 
because of its interactive and individualized character. 
Trainees inherently receive the most benefit from the 


Euhorial nature ol ve nee 


B. NEED FOR RESEARCH 

There are various ways to create and present instruction 
and training. Each way highlights different aspects of the 
teaching-learning process. One way depends on a "master" 
teacher. The teacher, working alone, creates material or 
chooses content and presents training directly to students. 
Some historical examples of great teachers are Buddha, 
Confucius, Jesus, Mohammed and Plato. However, a great 
teacher is very rare indeed and generation of original 
training ends with his or her death. 

A second way focuses on the student. The teacher 
provides an environment for learning but the student must 
take the initiative and is primarily responsible for 
learning. This has the distinct disadvantage of allowing a 
long or uncertain period of time to attain a specified level 
of skill or knowledge. Finally, the student "may not be 
able to get there from here" without intervention from a 
teacher or other external stimuli. 

Another way relies on "expert" logic. A problem is 
defined, a solution is proposed and implemented, data 
gathered from the resulting training process and the results 
analyzed. The analysis and feedback are used to make 
revisions for improving the training process. Once a good 


t 


"formula" for effective training is found, it can be 


replicated or adapted to other uses. AS a practical matter, 


this systems approach is applied to virtually all training 
programs. 

Instructional Systems Development (ISD) is a process 
which provides a general systems approach with multiple 
components, operating within a certain set of constraints, 
to produce an instructional system. [Logan 82:pp. 1-3] CBT 
is frequently generated with ISD. 

CBT has often been applied directly to existing training 
methods. Just as there is no "cookbook" solution for ISD, 
there is no "best" way to do CBT. As a consequence, for 
most existing CBT, instruction could be done satisfactorily 
without a computer. This is not to say that CBT is 
ineffective. In certain situations, CBT can be and is very 
appropriate. [NPRDC TR 84-54, 84:pp. 4-19] 

This research integrates computer-based processing and 
training objectives with Database Management System (DBMS), 
Decision Support System (DSS) and Expert System concepts. 
The focus is on CBT development and implementation, for the 
Navy's Maintenance Training Improvement Program (MTIP). CBT 
development has not adequately recognized the formidable 
record-keeping requirements of a typical military training 
program. Thousands of learning objectives must be 
developed, cross-referenced, tested and taught. 
Additionally, CBT streamlines record-keeping and facilitates 
the CBT process to provide management information for better 


evaluation and monitoring. CBT users need the ability to 


customize and adapt CBT to provide the best training 


possible. [NPRDC TR 84-54, 84:pp. 10-11] 


Pee bENEFITS OF STUDY 

Training innovations have been described as having an 
unusual three-stage life cycle. In the first stage, 
advocates of the innovation proclaim its usefulness and 
success. CBT may be considered such an innovation. In 
Stage two, indiscriminate use becomes widespread, often 
without sufficient development, planning or training for 
intended users. In the final stage, the innovation's 
shortcomings become apparent from experience. Criticism and 
skepticism grow, but come too late in the process to help 
and hasten abandonment of the technique in favor of another 
new innovation. The cycle begins anew. [NPRDC TR 84-54, 
eeep. 14] 

A benefit of this study is a contribution to breaking or 
modifying the cycle for CBI applied to training programs 
such as MTIP. CBT must be thought of as a combination of 
learning theory and computer technology. The best CBT; 
however, results from the development and recognition of CBT 
itself as a new technology. Ultimately, this improves the 
methods and processes for providing training. For example, 
naval aviation maintenance personnel use of MTIP with 
refined CBT would produce better trained and more qualified 


technicians. This results in less time spent for 


maintenance and improved quality in the maintenance 
performed. These factors will significantly contribute to 
higher sustained readiness for today's modern sophisticated 


aircraft, support equipment and systems. 


D. - sGORE. Oia sian, 

This research is applicable to related aspects of 
existing and future computer-based education, instruction 
and training. The study evaluates a limited military 
application for the use of CBT to further enhance MTIP. The 
training of enlisted naval aviation maintenance personnel is 
the target. Such an environment is characterized as having 
standardized procedures and uses easily verifiable data. 

The training is required for a large number of people and 
must be conducted frequently. A wide variety of students 
must finish training with a minimum level of skill and 
knowledge. 

Research is based on current books, periodicals and 
technical reports. Application is made to the military 
training environment. Recommendations based on the research 
are intended to support management policy-making to provide 
the best CBT for the military maintenance training 


environment with the resources available. 


E. THESIS ORGANIZATION 

Chapter II provides a brief historical perspective of 
CBT. Examples are given of potential future applications 
wom CBI. 

In Chapter III the fundamentals of database processing 
are described. Examples are given of how the Navy can and 
does use database management in CBT. The chapter concludes 
by enumerating several major advantages and disadvantages of 
database processing. 

Chapter IV explains features of Decision Support Systems 
(DSS). The attributes of the DSS major subsystems--data 
management, dialogue management and model management--are 
presented. Ways in which DSS can be applied to CBT and Navy 
training programs, especially MTIP, are described. 

The chapter also discusses Expert Systems used for 
training. The three major problems of using Expert Systems 
for knowledge acquisition are presented and a model of an 
Expert System used for training, TUTOR, is illustrated and 
described. 

Chapter V is about specific issues of CBT and the Navy's 
Maintenance Training Improvement Program (MTIP). 
Instructional Systems Development is presented as the 
preferred method of CBT introduction for MTIP. 

Chapter VI follows with conclusions and recommendations 
about the impact and effectiveness of CBT on the MTIP and 


Navy training programs in general. 


II. BACKGROUND 


A. HISTORICAL PERSPESGI i 

Early developments in Computer-Based Training (CBT) were 
in the realm of Computer-Managed Instruction (CMI). CMI 
served as the foundation on which Computer-Aided Instruction 
(CAI) was built. CMI provided an environment which promoted 
and fostered CAI development as more cost efficient while 
providing more effective learning. 

several reasons contribute to the CMI evolution as a 
foundation on which CAI developed to form the principle 
components of CBT. First, computer hardware was very 
expensive. Budgets did not provide funding for high start- 
up costs of computer equipment and facilities for CBT. In 
many Situations it was not economically feasible to replace 
conventional instruction with CBT or to initiate CBT to meet 
an unfilled training need. 

Additionally, computers were difficult to use without 
extensive training. Most people could not readily use 
computers and their use was not conducive to the direct 
learning process. Computer literacy was low. Thirdly, the 
software for CBT was very limited in quantity and the 
quality needed to build dynamic learning systems. 


Presently, sophisticated software is readily available which 
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memmuch more Supportive of the active, natural learning 
process for a large number of people. 

Several early systems were developed and have become 
well known. First among these is PLATO (Program Logic for 
Automatic Teaching Operations). It was originally developed 
at the University of Illinois and has now matured over two 
decades. It is currently marketed by Control Data 
Corporation. TICCIT (Time Shared Interactive Computer- 
Controlled Information Television) was developed by the 
Mitre Corporation and Brigham Young University. It was 
marketed by the Hazeltine Corporation beginning in 1976. 
Finally, AIS (Advanced Instructional System) is a system 
developed by McDonnell Douglas Corporation in conjunction 
with the Air Force Human Resources Laboratory. [Kearsley 
83:pp. 70-74] 

The U. S. Navy is regarded as a pioneer in early CBT 
research and development efforts [Hansen 75:p. 7]. These 
early efforts were in the area of CMI and later evolved and 
matured into the broader manifestations of CBT. [In the 
1950's and 1960's, the Navy, through the Office of Naval 
Research (ONR) (as it was known then), sponsored and 
directed CMI research by various university and other 
organizational research centers. The Naval Training 
Research Laboratory (NTRL) (as it was known then) was 
established. CMI was initiated at the Naval Air Technical 


Training Center (NATTC), Memphis in Millington, Tennessee. 


ie 


Although leading in development and research, the Navy was 
somewhat slower to implement early CBT than the Army or Air 
Force [Fletcher 86:pp. 183-197]. 

However, the Navy, as well as society as a whole, found 
itself surrounded by increasing technological 
sophistication. People were required to master more 
complicated subject matter and display more technical 
skills. Technology brought advances, i.e., improvements in 
job performance and conveniences; but it was less tolerant 
of ignorance or errors in judgment. Implementation of more 
advanced technological skills required higher standards of 
performance. CBT demonstrated a method of learning which 
reduced cost while increasing learning effectiveness. 

Technology increased productivity with a positive return 
on the investment of resources. Thus there was a 
"technology push" to advance and further improve the state 
of CBT. Through CBT technology, human capabilities are 
extended; they are "leveraged" to improve and expand human 
productivity and accomplishment. 

CBT differed from earlier learning methodologies 
predominantly in that it invoked active learning. A learner 
had to do something--answer a question, select a topic, ask 
for help, etc. Computer technology was combined with 
learning. It also provided the avenue for a person to learn 
interactively from a machine or a computer controlled 


system. Although training systems could be developed which 
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were passive, the most effective systems were those which 


were interactive. 


B. REASONS FOR COMPUTER-BASED TRAINING 

mae rationalesior CBI ais fairly stralehtforward. It has 
long been recognized and accepted that no two people learn 
at the same rate, i.e., learning abilities vary 
considerably. Each person can learn most effectively when 
the sequence of instructional material, the pace and mode of 
instruction and even the style of presentation are tailored 
to his individual capabilities and desires. CBT offers an 
Opportunity for highly individualized and selective training 
Meynch 7i:p. 4]. 

Almost all CBT applications are developed by the need to 
improve cost efficiency and learning effectiveness. While 
these are fairly general goals, they normally manifest 
themselves in more specific objectives [Kearsley 83:p. 2 
FE]. Some of these more specific reasons for CBT are 
illustrated in Table 2.1. These reasons are amplified 
individually and some potential future considerations are 
presented in the following pages. 

Poimproved Control 

Training can be made uniform and established at a 
desired level of standardization which is optimal for the 
organizational needs. Consider the case of a maintenance 


EGaimings course where trainees must spend time practicing 
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TABLE 2.1 REASONS FOR COMPUTER-BASED TRAINING 
[Kearsley 83:p. 2] 


Improved Control 

Reduced Resource Requirements 
Individualization 

Timeliness and Availability 
Efficient Use of Training Time 


Improved Quality of Training 


Discretionary Management of Training 


Change Agent 
Improved User Satisfaction 
Consistency 


Reduced Development Time 





repair of actual equipment malfunctions. Before students 
attempt the repairs, CBT can provide control to ensure that 
proper trouble-shooting techniques are taught and followed 
and that good diagnostic skills are developed. With a solid 
basis of theoretical and procedural training, trainees may 
further develop and refine their new skills with actual 
physical "hands on" training. The entire learning process 
is controlled and evaluation of learning accomplished is not 


based only on certain checkpoints along the way. The 
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process itself as well as the achievements are open to 
inspection and review. 

Computers also improve control by facilitating the 
administrative functions of the training system. These 
functions are in the realm of CMI. They may include 
registering students, assigning training schedules, 
generating and scoring tests, producing required reports, 
distributing instructional material and other necessary 
Ime tions. 

CBT can be used to monitor student progress. 
Students receive instruction which is intended to provide at 
least a minimum level of comprehension and result in 
improved job performance above a standard level. However, 
comprehension rates vary from person to person and subject 
to subject. Improved control can ensure that these 
differences are quickly identified and action taken to 
provide optimum scheduling and assistance to promote student 
Mastery of the subject material. 

fee auced sRhesource Kequirements 

An established CBT system can reduce resource 
requirements. However, significant, high start-up costs 
must be considered when implementing a new CBT system or 
pomvertine from a conventional training program to CBT. 
These initial costs must be weighed against the life cycle 


benefits to be gained from CBT, including future rapid 
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assimilation of state-of-the-art training improvements which 
may be obtained from later CBT developments. 

CMI may reduce resource requirements by 
necessitating fewer people and facilities to maintain 
records and administer a training program. This reduces 
some of the overhead costs of training. CBT is 
individualized and tends to reduce training man-months. 
Therefore, fewer instructors, classrooms and instructional 
materials are required for a given student load compared to 
conventional training. 

CBT permits higher student-to-instructor ratios. 
Routine aspects of training may be accomplished more 
efficiently by CBT and instructors have more time to help 
students with problems. The CBT system can, in fact, 
pinpoint specific problems of each student for the attention 
of the 1nstriletor, 

Another important reduction in training resources 
from CBT is the use of computer-based simulators and 
trainers instead of actual equipment, such as aircraft, for 
training. Hands-on practice with actual equipment is 
critical to some training programs. If equipment is 
unavailable, crucial trouble-shooting or diagnostic training 
may be diluted. Computer-based simulators and trainers can 
provide realistic practice without the full cost of actual 
equipment. Finally, computer graphics can provide a student 


with a broader understanding of equipment operation than is 
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hoestble with mere practice on actual static equipment in 
the training environment. 
Bee endividuallzation 

Individualization has always been a principle 
objective of CBT because of the efficiencies it brings to 
the learning process. Each student is allowed to learn at 
his own pace and in the fashion most suited to him. This 
results in substantial time savings to complete training 
(See Table 2.2). In conventional training, instruction is 
often geared to a set pace based on the aggregation of past 
student performance. Thus training time for each student is 
the same even though student learning abilities may vary 
widely. Individualization, as provided by CBT, allows 
Students to learn at their own pace and, thereby make better 
use of total training time. 

CBT systems also allow students some control over 
the sequence of instruction. While completing a training 
course, for example, students may spend more time with 
topics they find difficult to master and less time with 
topics they understand. CBT may also give students more 
diseretion in scheduling their training time. 

Students begin training with different past 
experiences, levels of skill and knowledge. With CBT, 
student skill and knowledge levels may readily be assessed 
at the beginning of training and during execution. An 


individualized training program may be created which best 
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matches the needs and abilities of the student and the 
requirements of the organization for which the training is 
conducted. Maximum benefit may be obtained from a training 
opportunity. CBT can aid the training manager's decision- 
making; more to come about that later. 

Individualization is important for remedial 
instruction conducted during or after training. The Navy 
frequently uses remedial training in "A" schools and as part 
of the Maintenance Training Improvement Program (MTIP). CBT 
can identify specific problem areas and provide assistance 
to ensure that each student completes training with an 
acceptable level of skill and competence. 

4, Timeliness and Availability 

CBT provides significant advantages over 
conventional classroom training by being able to provide 
training "on demand." Training can be provided at the right 
time and in the right place. Students need not wait for 
specific training to become available. Pertinent training 
can be made available exactly as required. 

Students typically are provided certain training 
periods when they are absent from their normal jobs. During 
these periods they may be given an entire syllabus of 
training for immediate and future needs. Unfortunately, 
much of the training for future needs is often forgotten or 


becomes obsolete by the time it is needed. CBT allows 
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maining to be provided in smaller, more directly applicable 
packages. 

CBT has the potential to provide training at remote 
sites. The training can come to the students instead of the 
students going to the training. Much of the travel time and 
expense for training can be eliminated. The training can be 
dispersed by providing instructional material to each 
requiring site or by distribution over a computer network 
mmom a central facility. 

Finally, students are able to learn about new 
equipment before it has arrived through CBT trainers or 
Simulators. Thus they are more proficient in operating the 
new equipment when it does arrive. Students may also be 
trained in new maintenance procedures and techniques as soon 
as they are introduced. 

Deeeeecticient Use of Iraining Time 

With CBT, trainees can obtain the same level of 
learning in less total time than with conventional 
instruction. However, CBT is not a training panacea. In 
the military setting, where large numbers of people need 
training frequently, it may be very appropriate [Dean 83:p. 
150]. For example, a study of the effectiveness of CBT in 
the military demonstrated time savings of about 30 percent 
based on findings for Air Force and Navy training [IDA P- 


irene /9:pp. 5, 83]. See. also Table 2.2. 
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More efficient use of training time is almost 
completely attributable to individualization. Thus non- 
computerized training could also be made more time efficient 
with more individualization but probably at higher cost. 
However, as a practical matter, it is virtually impossible 
to establish and manage a large individualized training 
program without the use of a computer. 

With a CBT system, an assessment of student skill 
and knowledge levels may be facilitated prior to and during 
a training course. Then a course of instruction may be 
promptly created and modified along the way to provide the 
Student with the most learning potential for the training 
time available. The student's time is not wasted going over 
subjects he already knows and understands. Rather, he is 
challenged with unknown material and can make better use of 
the training time. 

More efficient use of training time permits students 
to train but spend less time away from their normal jobs. 
Sometimes, the length of training is the criterion for 
determining if that training can be taken or not. CBT has 
the potential, under favorable conditions, to provide the 
most training for a specified time period or provide 
specified training in the minimum time. 

The efficiencies of CBT may require changes to an 
existing training system and organization. Conventional 


training systems built around fixed schedules must change to 
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take advantage of CBT. For example, when the Navy first 
introduced CBT, training assignments remained on a fixed 
schedule. Most students finished their training early and 
were assigned undesirable tasks, i.e., watchstanding, 
recruiting duty, mess cooking or clean-up detail, while 
waiting for their scheduled training time to end. The word 
quickly spread that finishing training early was not a good 
idea and training completion time actually increased to 
nearly match the scheduled time. The Navy soon changed the 
assignment procedure from fixed duration to a continuous 
method and the problem disappeared. [Kearsley 83:p. 10] 
6. Improved Quality of Trainin 

Quality of training is difficult to measure and 
prone to subjective evaluation. Criteria may be established 
by which to appraise quality and measure effectiveness. The 
measurements may be direct or indirect. Direct measurements 
are those which gauge training based on evaluation of the 
training process itself and not its effectiveness. Indirect 
measurements are those which gauge training quality based on 
evaluation of the results of training. Both direct and 
indirect assessments may be important. Direct results 
provide immediate indications of discernible deficiencies 
which are usually readily correctable. 

The ultimate measure of the quality of training is, 
however, its effect on improving subsequent job performance, 


an indirect and subjective measure of training quality. The 
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purpose of training must be to improve job performance and 
that must be held at the forefront of all training programs. 
The users of CBT must evaluate its impact on improving job 
performance. High quality training should translate into 
improved subsequent performance in an existing job or lead 
to high quality initial job performance. 

Orlansky and String have compiled approximately 30 
studies (48 sets of data) which report on the effectiveness 
of CAI and CMI in military training. These results are 
presented in Table 2.2. 

CBT provides several features which tend to improve 
the quality of training. In interactive, individualized 
instruction (such as good CBT), students spend a much higher 
percentage of time involved in the process and learning, 
i.e., Students participate more in training than in 
conventional instruction. Secondly, CBT can provide 
pertinent feedback to trainees and instructors. Feedback 
may reinforce positive accomplishment in the form of 
messages and new assignments. Instructors may also 
intervene with praise, recognition or provide special 
privileges. Feedback may also suggest corrective action for 
less than satisfactory progress thus ensuring that training 
meets at least the minimum of established quality standards. 

CBT may be used to improve job performance directly 
or indirectly. In the direct approach, students are trained 


in specific skills, for theic immediate jobes mia t one 
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example is inservice training such as the Navy's MTIP. Job 
performance is expected to improve as students are trained 
in specific skills for their immediate job situation. Job 
performance is expected to improve for a particular job 
role. Using an indirect approach, instruction may be 
provided which is more general purpose and more broadly 
affects job performance, such as remedial instruction in 
mathematics, reading or finance. This training may be 
helpful for many jobs and in the student's personal life as 
well. 

7. Discretionary Management of Training 

CBT may be more accommodating and flexible than 
conventional training to trainees and employers of trainees. 
These characteristics may lead to improved morale and 
PrOduecrrvity. 

Students have, within limits, the freedom to pursue 
training at their own pace and to fit their own preferred 
schedule. More time may be spent on topics which are more 
interesting to the student or which require more time to 
Master. The trainee may receive feedback to indicate areas 
requiring more study or to confirm perceived levels of 
training comprehension. 

Managers--those who supervise training programs and 
those who send trainees--are afforded more professional 
discretion in their positions. A training course can be 


Started at any opportune time. It is not imperative that a 
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class schedule be set and then quotas filled with students 
Mreeraining to be conducted. Courses of instruction can be 
easily modified or updated as valid needs arise with good 
database management as discussed later. Additionally, CBT 
provides the technology to provide training to the user 
Sees, if warranted, rather than requiring that training be 
conducted at a central training facility. Courses may be 
readily tailored to the needs of individual customers. 

With computer technology becoming ever more 
ubiquitous and widespread, CBT provides the opportunity for 
students to work with and learn on a computer system. As 
they learn, they also become familiar with a computer 
system. They may even be able to learn on a system which is 
identical to or similar to one they will use in their job. 

8. Change Agent 

CBT is a different way of presenting training than 
conventional classroom instruction. It is an alternative 
way to accomplish common or more ambitious goals. CBT can 
provide many benefits as described here. Incorporation of 
CBT in an organization will require managerial change as 
well as technological change to realize its maximum benefit 
Weearestey 85:p. 61 fF, and 83:p. 13-14, Seidel 8l:p. 211 
ff]. For example, application of a computerized Database 
Management System (DBMS) or Decision Support System (DSS) 
makes more sense with CBT than with conventional 


instruction. More about this in later chapters. 
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Note that change is made for organizational benefit 
and not merely for the sake of change. Change is made to 
provide a suitable environment for the new training system 
and an atmosphere in which it can function best. 
Effectiveness measures must be examined and re-evaluated. 
Instructor roles and functions must be made compatible with 
CBT. Users of CBT must learn how to use a computer. These 
are some of the most apparent changes driven by introduction 
OL FCEL. 

CBT allows changes in instructional materials and 
methods to be made quickly. There is no need to print and 
physically distribute changes to all concerned parties. 
Changes can be made in the computer system and are rapidly 
available to all users. If common CBT is conducted at more 
than one site, changes can be distributed and incorporated 
electronically. Potential system modifications may be 
tested in CBT by pilot, phased or prototype projects before 
being approved for system wide use. 

Because CBT is very accommodating to change, it 
promotes change. The wide variety of training 
presentations, i.e., text, graphics, trainers, simulators, 
encourages developers to create potential improved methods, 
material and equipment to deliver training. As these become 
proven and accepted, they can be implemented as constructive 


changes to a CBT system. 
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9. Improved User Satisfaction 


Instructors may find improved professional 
satisfaction in CBT because the system facilitates many of 
the administrative and routine functions required of them. 
More of their time may be spent helping students and 
monitoring student progress. It is also easier to keep 
training material current and pertinent. When compared to 
conventional instruction, CBT offers high potential to 
provide many more training resources to instructors. 

In the training environment, student aptitude and 
attitude are important considerations for successful 
learning. CBT can stimulate positive attitudes. Student 
participation in the learning is high, leading to higher 
self-satisfaction with learning accomplishment. Students 
can somewhat tailor the sequence and pace of training to 
their personal style and are more involved in the learning 
process. [Kontos 84:p. 4] 

CBT, especially graphics, trainers and simulators, 
can provide more realism to training compared to 
conventional classroom training, thus giving the student 
more of a sense of purpose and achievement. Rapid response 
time and feedback are CBT characteristics which improve user 
satisfaction with the system. Additionally, students can 
assess their progress during training and that creates a 
sense of accomplishment and satisfaction. Improved 


Satisfaction leads to increased motivation which strongly 
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complements any level of trainee aptitude. Together these 
tend to improve training productivity and reduce (possibly 
eliminate) training course failures. 
10. Consistency 

Human instructors can have an "off day" or miss a 
day because of illness or personal reasons. This interrupts 
a course of instruction. A substitute instructor will most 
likely not present the same material in the same manner as 
the originally scheduled instructor would have. With CBT 
more consistent training can be presented to trainees. 

During development, subject matter experts (SME) 
and training designers may work together to create material 
to cover a topic to meet course objectives. The consensus 
of these expert opinions can result in a computer-based 
course of instruction that will satisfy course objectives 
each and every time it is presented. 

ll. Reduced, Vevelopmenes ime 

Training materials often require modification as 
procedures, regulations or equipment change. In fact, out- 
of-date materials is a major cause of ineffective training 
and dissatisfaction with training. Authoring can be 
facilitated with CBT. Also, making revisions to existing 
material based on new developments and implementing them is, 
frequently, time-consuming and difficult. CBT streamlines 


the revision process. 
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12. Future Enhancements 

Computer hardware costs have decreased dramatically 
while capability has increased greatly. If future advances 
are to be realized, CBT must be established as a foundation 
on which to build. Two most likely future advances, which 
may already be found in very limited use, are discussed in 
the following paragraphs. 

a. Embedded Training 

Embedded training refers to integrating 

training functions directly into weapon, aircraft or 
maintenance equipment systems. Embedded training is a facet 
of CBT which is developed with the corresponding equipment 
or system as an integral part of it. An advantage of 
embedded training is that it is very specialized training in 
which a person is taught on and by the actual equipment or 
system he will be using. Secondly, the training should be 
of high quality since it is produced by the same source as 
the equipment or system and designed specifically for that 
equipment or system. Also, training may be conducted in the 
field and is not limited to training facilities; it is 
available wherever the equipment or system is located. 

b. Networks 

Communication networks will permit training to 

be readily distributed to many users. Small users, who 
would not normally be candidates for a CBT system, may have 


access to CBT via a network. Modifications to existing 


22 


training material and new training material may be easily 
incorporated into the CBT system with virtually no delay and 
are quickly available “Ge users. 

Networks allow for greater physical security by 
distributing duplicate programs and databases at various 
locations in the network. This may also permit the CBT 
system to function even though part of it is down for 


scheduled maintenance or other unplanned reasons. 


C. DISADVANTAGES OF COMPUTER-BASED TRAINING 

A major disadvantage of CBT is high start-up costs. 
These are generally one time costs for system development, 
facilities, hardware and software. As development is 
completed, equipment is procured, installed and tested. 
Additionally, the recurring or maintenance costs for the 
System must be considered for the life cycle. The benefits 
to be gained from a CBT system must be weighed against these 
COSEs. 

Rapid obsolescence may be another significant 
disadvantage. Computers, as instructional tools, appear to 
be at a rudimentary state of development [Bork 87:pp. 201- 
206; NPRDC TR 84-54, 84:p. 1]. Breakthroughs or rapid 
development in computer technology or CBT may make existing 
equipment or systems impractical or obsolete in terms of 


training effectiveness before their costs are recovered. 
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Unfortunately, computers can be manipulated or "gamed" 
by students. Part of this phenomenon has been formalized as 
Parkinson's Law which states, "Work expands so as to fill 
the time available for its completion" [Parkinson 57:p. 1]. 

Another disadvantage is that while student attitudes 
generally appear favorable toward CBT, that of instructors 
has been unfavorable in some cases (See Table 2.2). 
Contributing to this is the fact that instructor roles have 
not always been clearly defined nor expectations described 
momeGBl [NPRDC TR 82-45, 82:pp. 24-25]. 

CBT may also limit constructive interaction among 
students as they learn. As students learn individually, 
they may not advance and refine interpersonal skills which 
may be needed to work in more cooperative team roles for 


future jobs. 


Dee SUMMARY 

The chapter has presented a brief history of CBT 
including some examples of early systems. Mentioned are 
some of the conditions and requirements which promoted CBT. 
Several reasons are presented for using CBT; these reasons 
generally contribute to training cost reductions and/or 
improved learning effectiveness. Also, some additional 
potential future uses of CBT were suggested. Finally, some 
of the major disadvantages of CBT, such as high initial and 


maintenance costs, were described. 
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The Maintenance Training Improvement Program (MTIP) is a 
CBT system and embodies several of the characteristics given 
for CBT. CBT can allow MTIIP to be further improved and 
provide a foundation for future developments and 


enhancements as discussed later. 
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TII. DATABASE MANAGEMENT APPLIED 
TO COMPUTER-BASED TRAINING 

Data is a very important resource in a Computer-Based 
Training (CBT) system. Data includes facts and figures 
about reality from which information may be drawn and 
conclusions made. Information is selected data about 
specific circumstances or subjects which convey new meaning 
to a receiver. Knowledge is general understanding based on 
experience and education or training. Knowledge, either 
personal or expert, is applied to applicable information for 
decision making. [Harmon 85:pp. 30-31, Kroenke 87:p. 41, 
Wiederhold 86:pp. 78-79] The data held by a CBT system is 
fundamentally important to trainees, instructors and 
managers. The foundation on which CBT is built and 
functions is provided by data. 

A CBT system must be able to receive or gather data. 
There may be many sources of data. For example, data for 
training may be obtained from equipment manufacturers or 
Operators, or from other military organizations in the form 
of specifications or procedures. Data may be generated 
internally within CBT as the system is used. Users of CBT 
usually must provide some data to the system to be able to 
use the system. Other data stores, books, manuals, etc., 


may provide data to a CBI system. 
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Data which is input into the system must be subject to 
verification and periodic checks made to ensure the accuracy 
of all data. Once data is received and entered into the 
System, it must be capable of being accessed and processed 
in a reasonable time. Finally, data in a CBT system must be 
secure. Data should not be lost or easily stolen. 
Protection from unauthorized tampering, altering or copying 
must be provided. One common method of ensuring integrity 
is through data duplication. The capability to 
add, edit or delete data must be restricted to a few trained 
and authorized responsible individuals. 

Data and the subsequent information it provides are 
important to CBT users in slightly different ways. Students 
rely on data to provide information for training and 
learning. Thus, the student's knowledge base is expanded 
which should promote improved decision-making for better 
future job performance. Data is converted into information 
and incorporated into student knowledge before it is needed. 
The data is used for decision making outside CBT. Feedback 
May point to changes necessary in the data to provide better 
training. This student-data relationship is illustrated in 
Figure 3.1. 

Managers of CBT systems depend on data in a slightly 
different manner. For them data provides information 
directly for decision making about CBT itself. Information 


supplements existing knowledge in a more immediate situation 
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Figure 3.1 Student-Data Relationship 
[Wiederhold 86:p. 78] 


and through feedback becomes part of experience. The 
Manager-data relationship is illustrated in Figure 3.2. In 
the next section, a description is provided of how CBT 
provides a Management Information System (MIS) for the 
management of CBT. CMI is the component of CBT which 
includes the MIS functions. Figure 3.3 illustrates how data 
and databases fit into the scheme of a CMI subsystem. They 


serve a very central and crucial role. 
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Figure 3.2 Manager-Data Relationship 
[Wiederhold 86:p. 78] 






Feedback 


A. MANAGEMENT INFORMATION SYSTEM 
1, ~ De fi naitaon 

Management Information System (MIS) means different 
things to different people. At one extreme are those who 
view MIS as essentially equivalent to Electronic Data 
Processing (EDP). Thus data transactions are processed into 
scheduled reports. At the other extreme are those who 
believe that MIS should provide all the information which an 
organization needs to function effectively. Typically, much 
emphasis is placed on supplying management with information 


for decision making. [Sprague 77:p. 7] 
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Figure 3.3 Typical Components of a CMI Subsystem 
[Kearsley 83: p.. 25] 


Gorry and Scott Morton [Gorry 7l:pp. 69-79] describe 


a MIS framework for managerial activities. Managerial 


activities are categorized as strategic planning, management 


control and operational control. Strategic planning is the 


process of establishing objectives for an organization. 


Through management control, managers use judgment to assure 
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that resources are obtained and implemented effectively and 
efficiently to accomplish the organization's objectives. 
The third general function of operational control assures 
that system tasks are accomplished expediently and 
economically by carefully and previously delineated 
procedures. 

Decisions which are required of managers may also be 
classified as belonging to a continuum from structured or 
unstructured. Structured decisions are usually repetitive, 
routine and follow set procedures. Unstructured decisions 
are novel, complex, occur infrequently and follow no set 
procedures. Figure 3.4 illustrates the areas of information 
Systems generally thought of as MIS. The Structured 
Decision Systems (SDS) encompass nearly all of what has 
traditionally been called MIS. 

Gorry, Scott Morton and Sprague [Sprague 80:p. 10] 
reject the traditional definition of MIS for a broader, more 
practical view in which MIS refers to the whole set of 
systems and activities necessary to gather, process, manage 
and ultimately use data and information as a resource in an 
organization. The area of Decision Support Systems (DSS) 
applied to CBI, will be discussed in the next chapter. 

2. Database Management as a Foundation for MIS 

Data from which to derive information is essential 

to MIS. This data consists of data elements that comprise 


an organization's database. In the late 1960's, Head [Head 
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Figure 3.4 SDS and DSS in Management Information Systems 
(Gomny Flip. 974 | 
67:p. 23] explained that to accomplish anything useful with 
the database, it is necessary to provide organization and 
Bemucture. To do this, the information requirements of 
Management must be identified. The data elements must be 
identified in technical terms: how large they are, what 
their meaning is, where they are stored and how one can get 
at them. Finally, data relationships among data elements 
have to be identified. 

Differing levels and types of management require 
different types of information (see Figure 3.3). This can 
result in many specialized databases. In CBT, for example, 
instructors, training program managers and the Chief of 


Naval Education and Training (CNET) could each have their 
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own specialized databases. This hierarchical or 
"horizontal" design matches the database with the 
organizational level which is its primary user. Such a 
database is likely of very limited or no use to others in 
the organization. A proliferation of specialized databases 
may result in much duplication of an organization's core 
data. Databases have also been designed functionally or 
"vertically" to serve all levels of management with data 
about a particular subject. For example, in CBT, separate 
databases could be maintained for student records, test bank, 
learning activities (see Figure 3.3) and aircraft 
maintenance data for the Maintenance Training Improvement 
Programm Mili). 

Two particular advances in technology have aided MIS 
development. These are the development of large data 
Storage devices and interactive, real-time computer systems 
such as Database Management Systems (DBMS). The DBMS can 
produce a more responsive MIS and provide all levels of 
management with the functional information required from a 
common organizational database. The DBMS gives the 
organizational database design and structure, establishing a 
foundation for MIS as illustrated in Figure 3.5. The DBMS 
provides maintenance of the database as well as a 
communication network for gathering and modifying data. It 
also provides for inquiry to the database for flexible 


access and information retrieval [Sprague 77:pp. 8-9]. 
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BeeeeOOMPONENTS OF A DATABASE MANAGEMENT SYSTEM 

CBT employs an MIS which involves multiple users, 
multiple data files and requires multiple views of stored 
data. A Database Management System (DBMS) is a system of 
components which provides a more sophisticated approach to 
data management, storage and retrieval than traditional file 
systems. The five major components of a DBMS are considered 
applicable to CBT; they are: hardware, programs, data, 
people and procedures. 

1. Hardware 

With a computer installation such as CBT in place, 


there is very likely no need for additional specialized 
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hardware to implement database processing. A DBMS may, 
however, require more memory and storage components to 
accommodate its increased overhead. Several characteristics 
of hardware which would be advantageous to CBT would also 
facilitate DBMS. More main memory, faster Central 
Processing Unit(s) (CPU) and more direct access storage may 
be essential for the DBMS to function effectively. 

Recently specialized database machines have been 
introduced by manufacturers. These machines are special 
purpose computers whose only function is to manage the 
database. (See [Date 86:p. 603 ff] for a description of 
these machines.) Each CBT system would have to be analyzed 
independently to determine if such a machine could be used 
effectively in that system. 

2. software 

software refers to the programs used in the database 
processing system. Database requests and transactions may 
be processed in batch form or on-line by users at terminals. 
On-line processing uses a Communications Control Program 
(CCP) to receive and route requests. CCP provides 
communication error checking and correction. It coordinates 
activity among the various terminals and routes messages 
within the systen. 

Other types of programs are application programs and 
utilities. Application programs satisfy specific 


predetermined needs such as class lists, tests, learning 
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activities, etc. They must be developed for or tailored to 
each CBT system. Utility programs provide a wide variety of 
services. They may provide for generalized update and query 
of the database. Software locks are included to provide 
security for classified data. Others establish and maintain 
the database and can reformat and clean-up files for more 
effective operation and more efficient storage. 

A third type of program is the Operating Systen. 
This program controls other programs and controls the 
computer's resources. The operating system is usually 
designed for specific hardware and provided with it. The 
DBMS shields the user from any direct contact with the 
operating system. Figure 3.6 illustrates the programs 
involved in typical database processing. 

Be People 

People are an important part of the CBT DBMS. 
Obviously, there are users who employ the system every day 
for its intended function. As in other computer-based 
systems, there are operational and systems development 
personnel who are included in the DBMS. 

A very prominent position in the DBMS is that of the 
Database Administrator (DBA). Often referred to as a single 
person, this may actually be a group of people. Database 
administration is key to providing an environment for 
effective DBMS employment. A function of the DBA is to 


manage data activity. The processing rights of all users 
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must be determined. The DBA is the focal point for 
negotiating and assigning these rights. Different users may 
have different priorities. In a CBT system, for example, 
students and instructors are likely the most frequent users 
and have very high priority in keeping with the mission of 
Piearnine . 

However, required reports or management activity 
may occasionally pre-empt student and instructor use of the 
system. Tasks which are not mission critical should be 
scheduled for off-peak hours. Once these access rights are 
Originally determined or later modified, they must be 
documented and distributed to applicable users. The DBA 
must also formulate additional operating procedures for such 
items as concurrent database update, backup of the existing 
database and recovery of the database in the event of system 
failure. A CBT application where the role of the DBA is 
very important is the Navy Maintenance Training Improvement 
Program (MTIP) which consists of many databases each 
requiring sound, responsive management. 

The DBA is a central figure in the management of the 
DBMS itself. This is discussed further in the next section. 
Finally, the DBA is concerned with DBMS performance. A DBMS 
can be periodically tuned for more efficient operation in 
its present environment. This makes better use of existing 


resources to achieve system objectives. Evaluating current 
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performance helps determine which new software or equipment 
to procure for future requirements. [Kroenke 87:pp. 324-327] 
4. Data 

As stated earlier in the chapter, data is a very 
important resource for CBT as a foundation for the DBMS. 
Data in a database is integrated and shared. The database 
must be self-describing, i.e., it contains a description of 
the database structure. A self-describing database ensures 
that the data remains application program independent. A 
DBMS should provide a data dictionary, or user-accessible 
catalog which describes the data [Kroenke 87:p. 317]. 

An integrated database is more than a collection of 
data files. It is a collection of files with much 
redundancy eliminated, but includes designed relationships 
among the various file records. A shared database is one 
where different users, through the various application 
programs, have access to the same pieces of data. 

Data may be referred to in three additional ways in 
a database system. First is the logical view of all data as 
it would appear to humans in general. This is also known as 
the schema or conceptual view. Another view is the external 
or subschema perspective. This is a particular view by a 
specified user and provides different views of the same data 
depending on the particular perspective. There can be many 
overlapping subschemas for a given database and each is a 


subset of the general logical view. A third view is the 
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internal or physical view of data as it is actually re- 
presented. Figure 3./7 illustrates the three views of data. 

To implement the logical database, physical 
considerations are imposed on the conceptual model to create 
the physical database [Cesena 87:p. 5]. This is the view of 
data as it is stored and processed in a particular system. 
Application programs call the logical files which are stored 
in the physical database. The various views and 
relationships of data logical files as used by application 
programs in the CBT physical database is summarized in 
Figure 3.8. [Date 86:pp. 5-7, Kroenke 83:pp. 11-14] 

5. Procedures 

DBMS procedures are both system operating 
instructions and policies for system use. Procedures are 
established for system users and operators. Procedures may 
also be segregated as those for normal conditions or those 
for contingencies such as emergency, failure or recovery 
occurrences. As noted above, development of system 
procedures is a prime responsibility of the DBA. When 
establishing procedures, the requirements of all users 
should be considered with a view to the technical, 
operational and economic feasibility of each [Davis 83:pp. 
38-39]. Compromises or alternative solutions must be sought 
when conflicts arise. Procedures should not be at cross- 
purposes with one another and should promote optimal use of 


system resources for the benefit of all users. 
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[Cesena 87:p. 5] 
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C. MANAGEMENT 

As its name implies a DBMS is a management tool. The 
DBMS is very important for a MIS and has several components 
as discussed previously. Kroenke [Kroenke 83:p. 538 ff] 
describes three aspects of management associated with a 
DBMS. The first two, management of data base structure and 
management of database activity are implemented by the DBMS. 
The third is the management of the DBMS itself. Each of 
these has application for the CBT system. 

1. Management of Database Structure 

For database processing to function properly some 
standardization must exist. This provides a common 
reference for all users. A data dictionary which contains 
the names and structures of all records and files is helpful 
in this regard. Communication between users and DBMS and 
between users themselves will be improved. 

Database systems tend to reduce or eliminate data 
redundancy. However, some duplication may be required or 
desired to improve system performance or security. 
Redundancy which is permitted must be managed to ensure that 
recurring data is periodically reconciled. 

DBMS management must provide for configuration 
control. Database standards must be enforced. However, a 
process must be designed for making constructive changes as 
needed to accommodate a changing DBMS environment. As their 


requirements change, users should be permitted to suggest 
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appropriate changes to the structure. Information about the 
DBMS structure and changes made to it must be documented and 
distributed to users. 

fae anagement of Database Activity 

Data ownership is an important part of database 
activity. Who owns which data? How is data entered, 
accessed or modified? Who is responsible for the accuracy 
of data? Management must deal with each of these concerns. 

security must be provided for any classified data 
which may exist. Additionally, physical security must be 
provided against theft, loss, fire, etc. Recovery 
techniques and procedures must be established in 
anticipation of machine malfunctions, communication failures 
or incorrect system uSe. 

Database activity must include training for new and 
existing users. Initially, new users are to be provided 
with training to use the system correctly. As the system 
changes, procedures and policies must be modified and 
existing users informed and retrained if necessary. 
Management must enforce data activity policy and procedures 
for the benefit of all users. 

3. Management of the DBMS 

Effective operation of a DBMS must be planned and 
@omtrolled. in addition to periodic tuning of the systen, 
provision must be made for maintenance and modification. 


System use and performance should be evaluated as to how 
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well user requirements are being met. New products and 
services from industry and advances in technology should be 
monitored for application to the system to more economically 


meet user requirements. 


D. ADVANTAGES OF DATABASE PROCESSING 

A major advantage of database processing is that it can 
provide a CBT system with centralized control of its 
operational data. Centralized control allows more 
information from the same amount of data compared to a 
typical file system. Data may be more efficiently shared 
among legitimate users. With database processing, data 
redundancy may be significantly reduced and data 
inconsistency avoided thus providing more accurate data. 
Finally, database standards may be established and applied 
to an entire organization. With a database, one department 
or group can better manage the data through specialization 
and by providing a central point to implement database 


policies and procedures. 


E. DISADVANTAGES OF DATABASE PROCESSING 

Database processing has several potential disadvantages. 
First and foremost is the fact that it can be complex and 
expensive. High expenses result from the initial cost and 
additional hardware needed to support the system. 
Complexity implies high operating overhead which will 


actually slow down some simple sequential processes. A 
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complex system is more subject to degradation and failure. 

A system failure may affect most or all contained processes. 
With a large and complex system, recovery from failure is 
more difficult. It is also more difficult to provide 
security for a large and complex system which has many 


users. 


Pee SUMMARY 

The chapter began with a discussion of data as a CBT 
resource, which can provide information and knowledge. As a 
resource, data is of critical importance to CBT in general 
and the Navy's Maintenance Training Improvement Program 
Minter) as discussed later. CBT students, instructors and 
Managers use data to achieve their specific purposes. Data 
elements or the database serve as a foundation for the CBT 
Management Information System. Data is a critical resource 
of CBT and a DBMS offers the potential to manage it better. 
Managerial activities are classified as strategic planning, 
management control and operational control. Decisions 
required of training managers lie on a continuum between 
structured and unstructured. To meet these varying 
managerial requirements, specialized databases can be 
designed, which are often useful to very few users. DBMS 
permits necessary information to be obtained from existing 
databases and can provide a more responsive MIS for all 


levels of management and operational users. 
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The chapter concluded with a discussion of the major 
components of a DBMS and their functions. The success of 
CBT rests in large measure on the ability to manage and 
effectively use data. Database management was described as 
Management of data activity, data structure and the DBMS. 
Finally, several advantages and disadvantages of database 


processing were presented. 
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IV. INTEGRATION OF DECISION SYSTEMS AND 
COMPUTER-BASED TRAINING 
MeO ECISION SYSTEMS 

The decision-oriented approach to information systems 
attempts to identify the full range of alternatives and 
resources available to a decision-maker. Two types of 
decision systems will be considered as applicable to 
Computer-Based Training (CBT). The first is a Decision 
Support System (DSS) which supports decisions. The second 
moaman Expert System which makes decisions. These systems 
may be integrated for certain applications or employed 
independently. [Turban 85:p. 141] In either case, the 
decision action to be taken is usually left to the decision- 
maker. 

Mason [Mason 69:p. 81] directs attention to an important 
question. Namely, what is the optimal point of articulation 
or juncture between an information system and the decision- 
maker? That is, what are the roles of the information 
system and the decision-maker? Where does the information 
system begin and leave off and the decision-maker take over? 

Figure 4.1 illustrates three possible points of 
articulation which are represented by Database Management 
System (DBMS), Decision Support System (DSS) and Expert 


System. In Figure 4.1, decision-making is presented as a 
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sequence of events culminating in a course of action based 
on a decision being made. The sequence of activities is 
summarized as follows: 

1. Source--origin of data 

2. Data--used to provide information 

3. Inferences and Predictions--based on information 


4. Value and Choice--apply subjective criteria and 
ratings 


5. Action--make decision and proceed with course of 
action 


Identifying and defining these activities is key to 
designing an information system. 

Figure 4.la represents a DBMS which is described in 
Chapter III. The information system receives, classifies 
and stores data from which a decision-maker may draw 
information. The information system is designed to contain 
all the data necessary to provide information for the 
pertinent decision-making application. The database 
structure emphasizes economy with data duplication, 
inconsistency and omission minimized and superfluous data 
purged regularly to meet system objectives. This 
configuration presents facts and the decision-maker is left 


' This design is best suited for 


to determine "meaning.' 
managerial control situations where decisions are mostly 
objective and structured. Policies and procedures tend to 


be explicit and well established administratively. In CBT, 


it is this part of the system which provides scheduled 
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Figure 4.1 Information System and Decision-maker 
Relationship [Mason 69:pp. 82-86] 
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reports such as class lists, student schedules and other 
routine reports. 

An information system may extend beyond the activities 
of pure data collection, storage and retrieval to include 
the drawing of inferences and making predictions which are 
relevant for decision-making. This type of information 
system is a DSS, illustrated in Figure 4.1b. This process 
assumes certain relationships which are built into the 
system. An example is simulation models. The decision- 
maker is permitted to inquire "what if?" certain actions are 
taken based on assumptions which are considered valid. The 
system then provides results for all desired inputs based on 
the simulation model. The system does not evaluate outcomes 
and the decision-maker is left to assess outcomes and choose 
a preferred course of action. The next section provides a 
discussion of the major components of a DSS. An example 
from CBT, is the evaluation of an increase in course 
content. How does the change affect student schedules? Are 
the required resources available when needed? If course 
prerequisites change how will student loading change? 

A further evolution of the information system is an 
Expert System as pictured in Figure 4.lc. Such a system 
incorporates an organization's value system and criteria for 
choice. The design can be referred to as a decision-making 
system. The system will make a decision and the decision- 


maker then has an opportunity to reject the decision, accept 
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as is, or accept with modification and then begin a course 
of action. As its name implies, this process attempts to 
mimic the behavior of experts. The system incorporates the 
algorithms of experts to provide the best or optimal 
solution in a particular area of expertise. Expert systems 
are discussed in more detail later in Chapter V. 

As the information system develops through the phases 
discussed above, assumptions become intrinsic to the system. 
These assumptions may become "hidden" to the decision-maker. 
The underlying assumptions must, however, be in accord with 
the best judgment of the decision-maker for the information 
system to become a valuable management tool. Assumptions 
must be accurate and regularly updated. The decision-maker 
must develop confidence in the system for it to be employed 


effectively. 


EeeGAPABILITIES OF A DECISION SUPPORT SYSTEM 

DSS is not merely an evolutionary advancement of EDP and 
DBMS. It is a class of information system which draws on 
EDP and DBMS but certainly will not replace either. DSS 
requires a new combination of information systems technology 
to satisfy new requirements. The objective is to further 
improve the decision-making process. [Sprague 80:pp. 10-12] 
In designing a DSS, the components of the system must be 
identified and their interface specified. Three prime 


capabilities are crucial ina DSS. These capabilities are 
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provided by the data management, dialogue management and 
model management subsystems. [Sprague 80:p. 29] 
1. Data Management 

A collection of data maintained by a training center 
for planning, control, training and administration is its 
database (see Chapter III for further discussion). Sprague 
and Carlson [Sprague 82:pp. 223-225] suggest that database 
management is important to DSS for two reasons. First, a 
database and associated DBMS are needed as a prerequisite to 
building a DSS because building a DSS without them would be 
very difficult. They are basic building blocks for a DSS. 
The reasons for having a database prior to building a DSS 
may be summarized as follows: 


1. Simplifies collection and maintenance of data used 
by 22 svi 


2. Limits the functions and users that a DSS must 
Support 


3. Simplifies DSS design 


4. Reduces potential conflicting performance and 
security requirements 


5. Increases the opportunity for sharing data within 
a DSS while reducing data redundancy 


Having a database prior to building a DSS can reduce the 
cost of building a DSS while improving the accuracy of the 
data used ina DSS. 

secondly, data management is a major capability 
required of a DSS. Data management supports the memory 


requirements of a DSS. Data for DSS use is stored in 
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databases. Data management provides workspace for 
intermediate results and provides libraries for saving 
workspaces. Data management links related data, reminds 
decision-makers of operations to be performed and of data 
that should be considered. 

Data management includes the ability to reduce or 
abstract desired information from large amounts of data. 
For example, in evaluating a course of instruction or a 
training session, a training manager may want to review the 
prior training of a group of students who completed the 
course in question. With data reduction, he should be able 
to obtain the required information without going through 
each student record. 

DSS should provide varying levels of detail. 

A manager on the staff of the Chief of Naval Education and 
Training (CNET) may need information about how Aviation 
Machinist's Mate (AD) students have been graded over the 
past year. On the other hand, an instructor may wish to 
know how a particular class of AD's has done on tests during 
an ongoing course. 

2. Dialogue Management 

Dialogue is the interface between the decision-maker 
and the DSS. It is interactive and determines the user 
friendliness of a DSS. Dialogue management frequently 
includes the largest percentage of computer code in a DSS 


and is most often modified [Sprague 82:p. 217]. Dialogue 
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Management provides and manages communication between the 
decision-maker and a DSS. The form of communication often 
includes graphics in addition to text. 

There are several dialogue styles which may be found 
in a DSS. A common form is the question and answer 
dialogue. The DSS presents the user with a question, the 
user answers the question and the session continues until 
the DSS produces the answer to support decision-making. 
These dialogues are most helpful to inexperienced or 
infrequent users who are not familiar with the DSS. 

Another dialogue style is that of command language. 
Certain commands invoke DSS functions. For example, a CBT 
Manager may enter a command such as PLOT LEARNING RATE and 
obtain a plot showing a student or class learning rate for 
all training course modules completed or in progress. 

A third common dialogue style is the menu. The user 
is presented with a number of alternatives from which to 
choose. Selection is made with a keyboard or another input 
device such as a mouse or light pen. There are also very 
limited applications of making a selection by touch-screen 
or voice instruction. 

It is very likely that a dialogue management 
component will mix or combine more than one dialogue style 
in any DSS. The objective is to keep the DSS compatible 


with the needs of the decision-maker. The user may actually 
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be able to choose a style of communication to suit his 
preference. [Sprague 82:pp. 199-205] 
35. Model Management 

It is the model subsystem that gives decision-makers 
the opportunity to analyze a problem fully by developing and 
comparing alternative solutions. Decision models may be 
embedded in the DSS with the database capability as the 
integration and communication mechanism between them. The 
dialogue subsystem gives users direct control over the 
operation and manipulation of models. This link provides 
the ability for true interactive modeling. 

To serve DSS needs, the model subsystem must support 
decision-making activities such as projection, deduction, 
creation and comparison of alternatives, optimization and 
simulation. To provide these capabilities, a library of 
models comprising a model base must be established. The 
model base may be large or small and includes permanent 
models, ad hoc models, user-built or commercially obtained, 
"off-the-shelf" models. Models are included for 
operational, analytical and strategic decision support. 

Some models are used as "building blocks" to support the 
construction of other models. 

The comprehensive set of models required for 
decision support has led to the development of a Model Base 
Management System (MBMS). There is a pervasive analogy 


between the database with its DBMS and the model base with 
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its MBMS. The MBMS facilitates several general functions 
which are very important: 


1. Generate--create new models quickly and easily; use 
model “building blocks" 


2. Update--change data in a model without restructuring 
the model 


3. Restructure--redefine model in response to changes 
in the modeled environment 


4. Catalog--categorize and maintain a wide range of 
models supporting all levels of management 


5. Integrate--access and link appropriate models 


6. Report--provide information for desired decision 
Support 


Through the dialogue component, the decision-maker 
has the ability to interrupt the running of a model, run 
model segments and change sequence or parameters. Model 
management ensures that models draw inputs and parameter 
values from the database or databases and returns output to 
the applicable database(s). This promotes accuracy and 
consistency in model generated output for decision support. 


[Sprague 80:pp. 22-24, 82:pp. 257-272] 


C. PERFORMANCE OBJECTIVES 

Sprague [Sprague 80:pp. 18-20] delineates several DSS 
performance requirements. No particular DSS may be able to 
meet all requirements. DSS performance depends on the task, 
the organizational environment and the decision-makers 


involkved: 
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A DSS should provide support primarily for semi- 
structured and unstructured decisions. For these decisions, 
databases and traditional information systems may be 
inadequate. DSS are designed to provide support for these 
hoonais Of problems. 

A DSS should provide support for managers at all levels. 
CBT managers from CNET to course instructors and training 
managers should find support from DSS to solve problems in 
their professional realm. 

A DSS should support decisions which are interdependent 
as well as those which are independent. Many decisions must 
be made in consonance with other decisions made by others or 
gproups. The following decision types should be considered: 

1. Independent--a decision-maker has full 
responsibility and authority to make 
a decision 
2. sSequential--a decision-maker makes a decision or 
part of a decision which is then 
passed on to another decision-maker 
3. Pooled--a decision results from negotiation and 
interaction among decision-makers (A Group 
Decision Support System (GDSS) is a 
specialized DSS) 

A DSS should support all phases of the decision-making 
process. Herbert Simon [Sprague 80:p. 19] characterized 
three main steps in the decision-making process as 
intelligence, design and choice. Intelligence is monitoring 


or searching the environment for conditions calling for 


decisions. Raw data is gathered, processed and examined for 
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clues that may identify problems. Design is creating, 
developing and analyzing potential courses of action. 
Choice is selecting a course of action and implementing it. 

A DSS should support a variety of decision-making 
processes. Each CBT manager has an individual cognitive 
style and approach to problem-solving for his specific 
functional domain. There is no one best decision-making 
method. A DSS should be process independent and user 
controlled and directed. 

A DSS should be easy to use. It should not further 
complicate the decision-making procedure. The user must be 
comfortable with a DSS and develop confidence in it for it 


to become a valuable and convenient management aid. 


D. SEXPERT Sores 

Expert Systems may lend considerable aid to the training 
process as well as its management. Although this technology 
appears to be in its infancy, there seems to be much 
potential to significantly improve and enrich the training 
process and the quality of training provided by CBT. 

An Expert System is a Knowledge Base Management System 
(KBMS) which is particularly useful for dealing with a 
diagnostic or other interpretative tasks [Mylopoulos 86:p. 
4]. Such a system attempts to mimic top human experts and 
embodies knowledge of a particular limited application area 


[Turban 85:p. 138]. 9ethe rolemetetite Expert woystem in 
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feemtsion-making is extracted from Figure 4.1] and illustrated 
im Figure 4.2. 

Expert Systems have evolved from earlier Management 
Information Systems (MIS) and often interact with them. 

For example, DBMS provide information to Expert Systems in 
the same fashion as a human expert would use a data source 
such as a book or library to obtain factual information. 
Expert Systems may also rely on the model base of a DSS to 
provide established mathematical and forecasting models to 
lend support for system decision-making and expertise. 
[Turban 85:pp. 145-146] However, there are significant 
differences between a DSS and an Expert System as 
Mmeustrated in Table 4.1. 

According to Bevan and Wetherall [Bevan 87:p. 21], the 
process of building the knowledge base associated with an 
Expert System is long and complex but contains very valuable 
information. Most Expert Systems are designed to be used in 
a consultative manner by non-experts to perform a task in 
the narrow area of expertise. In the consultative role, 
Expert Systems may aid a technician in equipment fault 
isolation and suggest appropriate corrective action. 
However, Bevan and Wetherall suggest that the idea of using 
the same knowledge base in a training role is very 
Giermactives 

The Air Force Human Resources Laboratory (AFHRL) has a 


research program to develop and demonstrate the technology 


67 


Prediction Choice 


Source and and Actiom 
Inference Value 





Information System (Expert System) Decision- 
maker 


Figure 4.2 Information System and Decision-maker 
Relationship for Expert System 
[Mason 69:p. 86] 


for an Intelligent Maintenance Advisor. The Maintenance 
Advisor would serve as an "expert" for both consultative and 
training roles. As envisioned, the Advisor would be a skill 
integrator by using and capturing the corporate memory of 
skilled technicians. This knowledge would be used in turn 
to provide trouble-shooting training for novice technicians. 
[Richardson 86:pp. 47-50] 
1. Varieties of Knowledge 

Knowledge may be classified in several ways. As 
illustrated in Chapter III, knowledge may be comprised of 
the complimentary ingredients of formal training and prior 
experience. As a result of formal study, knowledge is 
gained which is composed of definitions, axioms and laws. 
Successful trainees emerge from courses with a firm grasp of 


the terms, procedures and laws which constitute the formal 
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oS Expert System 
Objective: Assist human Replicate (mimic) 
human 
Who makes the decision? Human chooses Human considers a 
from options recommendation 
Major orientation: Decision making Transfer of 


expertise 


Problem area: Complex, wide, Narrow domain 
integrated 
Query direction: Human queries System leads 
system human 
Database: Factual Procedural and 
knowledge Factual 
knowledge 


theories and accepted principles of their training 
discipline. This type of "head" knowledge can readily be 
described but has yet to be applied in a practical way. 
fiarmon 85:;p. 30] 

This kind of knowledge is generally well-structured 
and is amenable to the formulation of general principles for 
correct behavior. Spiro states that it is this knowledge 
domain in which past educational practice has had the most 
success and cognitive psychology and artificial intelligence 


have made their greatest advances. [Spiro 87:p. 286] 
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Knowledge may also be compiled by personal 
experience or by learning from a mentor. The student learns 
to rely on rules-of-thumb to perform specified tasks or to 
solve simple problems. This type of knowledge is referred 
to as heuristics. [Harmon 85:p. 31] Knowledge of this type 
may be relatively ill-structured. It may be difficult to 
define a single set of attributes for all cases. 
Interactions between various features in complex problems 
proliferate and may be highly dependent on one another. 
General principles are insufficient to solve individual 
cases satisfactorily, resulting in problems which are very 
diffictlt to decipher and resolve. [Spiro 87;p-=eoo 

2. Expertise 

How do experts solve problems? They use expertise 
gained about a subject to solve problems in that realm. An 
expert is an individual who is recognized as being able to 
solve a particular type of problem that most other people 
cannot solve as efficiently or effectively. Experts are 
able to perform successfully because they have acquired a 
large amount of domain-specific knowledge--expertise. 

Expertise may be gained entirely from experience or 
wholly from education without the benefit of formal 
training. However, most experts, including pilots, doctors, 
accountants and technicians, begin by studying their 
specialty in school. They learn principles and theories 


which are considered as basic to their profession. They 
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Pomceainue by paining experience through working in their 
professions and perhaps, working with a mentor. In this 
way, they gain expertise and may become regarded as expert 


in their profession. [Harmon 85:pp. 31-32] 


E. EXPERT SYSTEMS AND COMPUTER-BASED TRAINING 
Experience-Consolidation system is a term which 
describes an Expert System intended primarily for training. 
These systems "telescope" aspects of the process by which 
practical field experience leads to the acquisition of 
expertise. Thus an expert methodology is scrutinized to 
determine and then teach its essential characteristics. 
These systems attempt to teach intuition and give people the 
ability to think correctly for themselves without explicit 
guidelines in particular realms. Experience-Consolidation 
Systems attempt to shorten the time required for the would- 
be technician to gain experience from years to months or 
days. These systems attempt to teach what has not been 
taught conventionally and has traditionally been left for 
"on-job-training" (OJT) or as a skill to be gained 
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through “field experience." [Spiro 87:p. 285] 
Experience-Consolidation systems most closely resemble 

the Computer-Aided Instruction (CAI) form of simulation. 

Simulation does require active participation by students and 


promotes learning by doing. However, there are significant 


differences between simulation and Experience-Consolidation 
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systems. Simulation usually does not convey all the 
multivariate attributes of a complex case. It is not 
concerned with the effective acquisition of multi-patterned 
information. And, simulation cannot rapidly present a large 
number of cases for consideration. Experience-Consolidation 
Systems and simulation may be used to complement each other 
in a training Environment. | spirome’ -pee2o || 
1. Implementation 

As noted above, the idea of using Expert Systems in 
a tutoring role is very attractive. Is this possible? Is 
it practical? Expert Systems for training or Experience- 
Consolidation systems attempt to shorten considerably the 
time required to acquire large amounts of experience and 
expertise and then pass these on to trainees. The process 
must safeguard the essential characteristics of the 
expertise so that it remains relevant. 

A dilemma is obvious. In the real world, these 
essential characteristics themselves hinder shortening of 
expertise acquisition time. First, actual cases occur over 
long duration. Also, the amount of information to be 
gleaned over time is large and by definition, ill- 
structured. The information cannot be arranged (chunked) in 
hierarchical fashion and there is simply too much to 
remember. Finally, cases happen in haphazard order without 
regard to instructionally useful arrangement. [Spiro 87:p. 
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Spiro |SplroeoT sp. 267 LE |ediscusses each of these 
problems and suggests ways in which they may be overcome. 
To use this technology successfully, a training program must 
solve the challenges presented by these difficulties. The 
following sections present several of these ideas. 

a. The Problem of Time 

The experience needed to gain acceptable 

expertise may last for years. This problem is addressed by 
gathering many cases and extracting from them the most 
significant multivariate information needed to fully 
characterize a case's environment. For the Navy Maintenance 
Training Improvement Program (MTIP), this could be 
accomplished by gathering many skilled technicians, regarded 
as experts in the applicable field, and conducting 
interviews or making observations. Why are exhaustive 
representations important? The need is to richly and fully 
present the context in which the decision factors occur. A 
principle of Experience-Consolidation systems is that in 
ill-structured domains, contexts are taught to teach 
concepts. Situations (cases) are thus presented from many 
points of view. 

b. The Problem of Information Overload 

This problem is the most difficult. The 

challenge is met by using specially designed graphic aids 
which are tailored to simultaneously present complete sets 


of complexly entangled patterns at a glance. This method 
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taps into everyone's finely developed perceptual processing 
system. Experimental psychologists have established that 
humans have an extraordinary ability to remember many 
briefly presented visual scenes. The message for CBT is 
clear. High quality graphics capability is essential for 
this type of training. This powerful ability of human 
thought is employed to encode the manifold features of ill- 
Structured conceptual situations. These situations can only 
be understood and comprehended by simultaneous appreciation 
of multiple, partially overlapping interrelationships. 

This perceptual solution to the information 
overload and memory problem provides an optimal 
representation for conveying pattern and context 
information. A perceptual view comprehends many things 
juxtaposed. In a short time, many coexistent parts are 
viewed in one field of vision. Consider all the information 
absorbed simultaneously by viewing the image of a face or a 
landscape. People readily perform on stored perceptual 
information to detect patterns of resemblance and 
dissimilarity. Experience-Consolidation systems use the 
computer to tap into a powerful processing system of humans. 
This system is well suited to processing large amounts of 
intricately entangled information in ill-structured 


knowledge domains. 
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c. The Problem of Arrangement 

An arrangement system must be designed to 
maximize the retention of robust case information in order 
to highlight case information which may produce application 
difficulties. The system must be able to distinguish cases 
which appear similar but produce different outcomes and 
those which seem unrelated but produce similar outcomes. An 
arrangement system must also classify cases so that many 
complex resemblance patterns within the same outcome 
Category can be learned. The power, speed and flexibility 
of a computer are needed to make a functional system. 
Highlighting multiple connections and similarities between 
cases, permits de-emphasis of the retrieval of precompiled 
diagnostics in favor of diagnostic building, to fit the 
needs of a given situation. If training is to transfer 
expertise, the flexibility of knowledge representation is 


erucial. 


Zee steineelLingent Computer-Aided Instruction 
aye Mode dl 


An Expert System which is employed for training may 
be referred to as an Intelligent Computer-Aided Instruction 
(ICAI) system. According to Bevan and Wetherall [Bevan 
87:p. 21], with this technique, knowledge bases are used to 
represent the teaching domain and to govern the operation of 
the instructional system. There are two advantages to this 


amp Goach: 
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1. The operational characteristics of the system may 
be easily modified by using a rule base to govern 
the operation of the instruction system. 

2. Isolation of domain dependent rules from 
instructional rules permits the creation of a 
generalized ICAI system which can be used for a 
variety of teaching domains. 

This is seen as a way to capture the knowledge base 
of an Expert System for training and provides a means 
whereby operational staff can observe the performance of an 
Expert System. Such a system has been created and is known 
as TUTOR. 

A model of the TUTOR System is illustrated in Figure 
4.3. In simplified terms, the system makes a current 
assessment of a student's ability and knowledge from the 
Student Model. The Teaching Generator then decides what 
kind of interaction (exposition or problem to answer, for 
example) to give the trainee and selects a particular topic 
to present. The Administrator presents the situation to the 
student through the Natural Language Interface and to the 
Domain Expert System for the correct solution. Finally, 
each interaction with the trainee is recorded in the Student 
Model, so that the Teaching Generator may choose what to do 
next. Note that the TUTOR is an Expert System in its own 


right. Its major components are more fully described in the 


following paragraphs. [Bevan 87:p. 22-23] 
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Figure 4.3 Intelligent Computer-Aided Instruction 
System Model (TUTOR Bevan s7ep. 23 } 


ae Administrator 
The Administrator serves as the nerve center of 
the system. It provides general management and control over 
System resources. It serves as the primary connection and 


main interface between components of the system. 
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b. Domain Expert System 

This component contains the domain specific 
rules as they would appear in a consultative Expert System. 
It provides information to check student answers and 
responses. An explanation facility should be included to 
provide trainees with the reasons for correct answers or 
responses. 

c. Natural Language Interface 

Compared to a consultative system, this 
component is more important for the training system. A 
teaching system will require more two way communication 
between student and machine. Sometimes the student will be 
controlling the dialogue such as when seeking further 
explanation about subject material, at other times the 
system will be presenting material and controlling the 
dialogue. A fairly sophisticated Natural Language Interface 
is needed to give a wide range of users freedom in 
expressing their responses or answers to problems. 

Another requirement of the Natural Language 
Interface in a generalized system is that it be able to 
modify the vocabulary when the subject domain is changed. 
This is important for technical training subjects and to 
accommodate unique military terms. 

The U. S. Navy has sponsored research to develop 
an "intelligent" interface to CBT. The interface would be 


capable of responding to questions and answers in free form. 
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Humans could communicate with computers in English without 
having to learn a special computer language. Furthermore, 
the knowledge base used for response generation could be 
automatically expanded as trainees and course administrators 
interact with the system. [Bryant 88:pp. 199-206] 
d. Student Model 

The Student Model maintains a record of a 
student's knowledge of the subject domain. It may contain 
information about a student's preferred learning strategy 
and level of expertise. Levels of expertise may be 
categorized into several classes such as novice, competent 
but needing revision, skilled or expert, for example. 

e. Teaching Generator 

The role of this component is similar to that of 
a human tutor. It provides initial, corrective and remedial 
training on individual topics in the syllabus. To determine 
trainee comprehension, it generates problems for the student 
to solve consistent with the student model and the 
curriculum material as presented. It determines the nature 
of problems for consideration and the method of 


presentation. [Bevan 87:pp. 22-25] 


ee OUMMARY 
The chapter began with a discussion of DSS and Expert 
System applications to training. The major subsystems of a 


DSS--data management, dialogue management and model 


ne 


Management--were described. Several performance objectives 
were discussed. A DSS must be easy to use, promote user 
confidence and not further complicate decision-making. An 
effective DSS will accommodate the needs of all levels of 
training management. Also, a DSS should provide assistance 
with both independent and interdependent decisions which 
lead to effective and efficient use of training resources. 
This chapter portrayed the Expert System as an 
powerful vehicle for training. Expert Systems have 
traditionally been relied on in a consultative manner to 
provide solutions for specific problems in the realm of 
expertise. The knowledge encapsulated by these systems 
makes them attractive for technical training. However, 
several major difficulties are apparent. The problems of 
time, information overload and arrangement are presented. 
Nevertheless, Expert Systems have been applied to 
training programs. A model of one such implementation, 
TUTOR, is illustrated. The system itself functions as an 
Expert System. It has five major components--Administrator, 
Domain Expert System, Natural Language Interface, Student 
Model and Teaching Generator. The roles and functions of 
each component are discussed. Although in its infancy, this 
technology may provide significant CBT potential with 
specific application to the Maintenance Training Improvement 


Program (MTIP) as discussed in the following chapter. 


80 


VV. COMPUTER-BASED TRAINING APPEITED TO THE 
MAINTENANCE TRAINING IMPROVEMENT PROGRAM 
pee LNTRODUCTION 

As described in previous chapters, the U. S. Navy has 
had a long association with various aspects of Computer- 
Based Training (CBT). In recent years, the Maintenance 
Training Improvement Program (MTIP) has become very 
important in the training of naval aviation maintenance 
technicians. For the MTIP, most data for learning and 
student information is computerized. 

MTIP is a training management system and a method of 
inservice training. The program is a command responsibility 
which attempts to identify maintenance performance 
deficiencies at the command, work center and individual 
technician levels. As deficiencies are identified, 
refresher or remedial training is conducted to provide 
correct technical knowledge and to improve job skills. [NAMP 
86:pp. 12-7 and 12-8; CNAP 86:p. 1] 

The primary means used to identify knowledge and skill 
deficiencies is through a testing program which includes 
standardized question and answer banks from which 
computerized tests are generated. The tests are intended to 
examine the knowledge and skill of maintenance technicians 


compared to their peers and standards established for each 
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occupational specialty or rating. Test questions are to be 
prepared by Subject Matter Experts (SME) and require 
periodic validation. [CNAP 86:p. 1] 

MTIP has been widely implemented and provides units with 
some flexibility to direct their own inservice training 
program and compare the outcomes to established standards. 
Although use is widespread, acceptance has been less than 
desired. The program is seen as administratively burdensome 
and not producing intended results, such as, significantly 
improved maintenance technician knowledge and skill levels. 
It is the thrust of this thesis that application of CBT 
technology, as discussed in previous chapters, may provide a 
means to improve the current MTIP and lead to future 


enhancements. 


B. COMPUTER-BASED TRAINING TECHNOLOGY FOR MTIP 

CBT technology involves much more than the physical 
components that are visible and obvious. It encompasses 
methods, techniques and procedures that result from 
tradition, regulations, vision and available resources. 
Technology may be divided into "hard" components (physical 
devices) and "soft" components (methodologies). While those 
components may be identified individually, they are closely 
intertwined in actual technology applications. When 


evaluating a training system, the interaction among 
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components must be considered as well as the components 
themselves. [Kearsley 84:p. 3] 

The MTIP is conducted mostly by aviation squadrons, 
Fleet Readiness/Replacement Aviation Maintenance Personnel 
(FRAMP) units, Naval Air Maintenance Training Group 
(NAMTRAGRU) units, Navy Engineering Technical Services 
iets) or Contractor Engineering Technical Services (CETS). 
This training is primarily passive and relies largely on 
static media such as printed technical manuals, workbooks 
and slides. There is also some use of dynamic media. This 
includes films and video. The use of an interactive media, 
i.e., a computer, for actual training does not exist in the 
MTIP. Computers have been used to maintain databases and 
administrative records. CBT does provide the capability for 
active learning with computers. 

1. Database Management 

Chapter III describes why data is a very important 
resource ina CBT system. It is important because of the 
information which becomes available to both students and 
managers. This is certainly true for MTIP and is an area 
where significant progress could readily be made. 

The MTIP has been perceived as producing too much 
inaccurate, misleading or wrong information. Much of this 
problem relates to the system development and is discussed 
later under Instructional Systems Development (ISD). 


However, an integrated computer-based system could be more 
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responsive to the users. It is now very difficult and time 
consuming for users to provide corrective input to the 
existing system. 

A Database Manager, similar to a Database 
Administrator (to use database jargon), should be 
established for each MTIP database. The Database Manager 
should have the power to control that database and not 
merely coordinate or monitor activity. Users should be able 
to provide change inputs quickly and preferably on-line. 

The Database Manager should be able to promptly evaluate the 
inputs and make changes as necessary. Proper evaluation of 
change recommendations will require the many resources 
including those of Naval Air Systems Command (NAVAIR), CETS, 
NETS and occasionally the commercial contractors of the 
System, aircraft or equipment involved. The Database 
Manager should be able to determine a proper solution for 
change recommendations and provide rapid response to a user 
and alter the database as necessary. 

The database must be very large to encompass the 
scope and depth of all data needed for complete maintenance 
training for even one type of aircraft. If the database is 
too limited students quickly learn the "right" answers to 
particular questions. As this "gouge" is passed to others, 
the MTIP looses effectiveness. A more advanced system 
should be able to produce differing questions and answers on 


the same topic. 
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The MTIP database could store appropriate lessons or 
lesson guides for MIIP topics in addition to question and 
answer banks. It has been difficult to coordinate training 
with established testing. CBT could improve the 
interconnection between training and testing materials. The 
training materials could be generated as they currently are 
by FRAMP, NAMTRAGRU and squadron personnel and be 
incorporated into the MTIP databases. 

With the right equipment, training could be 
conducted at the squadron without having to schedule 
training sessions with another activity. Training could 
also be conducted on a more individual basis so that entire 
work centers would not have to be absent for training at the 
same time. Individualized training has other benefits which 
are discussed later. Additionally, CBT could better 
facilitate remedial training sessions at a FRAMP or 
NAMTRAGRU. This would provide better coordination and 
improved use of training resources over a manual system. 

2. Decision Support 

Chapter IV describes the fundamentals of a general 
Decision Support System (DSS) and provides some specific 
applications to the CBT environment. DSS are created to 
provide assistance for semi-structured and unstructured 
decisions. For the MTIIP to be a true Maintenance Training 
Improvement Program, it must assist the training manager in 


decision-making to further improve and refine the MTIP., 
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For example, based on MTIP data, managers should be 
able to develop student profiles and be able to classify 
students into desired categories such as novice, apprentice 
or master technician. A DSS could provide an optimal 
training plan for each category of trainee. With 
information of this type, remedial training could be 
developed to better suit the needs of individual students. 
This could provide more stability and direction to a 
training program. 

A DSS function can also lead to better resource use. 
Training resources of several activities are now available 
to the MTIP. A centralized, responsive system could aid the 
decision-maker in the most efficient coordination and 
scheduling of those resources with user need and priority 
considered appropriately. Additionally, usage data of the 
various resources would be available to support training 
managers in decisions about redistribution of existing 
assets or further development and procurement of new 
hardware, software or training technology. 

A DSS can also serve to give indications of overall 
MTIP effectiveness. As criteria are incorporated, 
information on various aspects of effectiveness can be used 
for evaluation. "What if?" evaluation may be done for 
squadron, functional wing or type command changes. 
Appropriate corrective action may be initiated to improve 


effectiveness. 
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Mov oseacpakreon stnestu ly eemuct else data which is 
easily available and obtainable. It must be designed to 
produce output which is meaningful, timely information, not 
just data. The DSS must focus on the where, what and when 
of technical training for the various specialties or 
mernes. [Ruck™86:pp. 47,54] 

tee Expert oystem 

Expert Systems have typically been considered useful 
For consulting or as advisors. In these roles, the Expert 
System is relied upon to provide expertise for a specified 
topic. As described in Chapter IV, the idea of using an 
Expert System knowledge base for training is very 
attractive. Attempts have been made to convert or develop 
effective training tools from consultative Expert Systems. 
Wemancey 8/7:pp. 1 ££, Richardson 86:pp. 47-51] 

There are several possible uses of Expert Systems 
for future MTIP development. Expert System may be able to 
impart expert knowledge to maintenance technicians. This is 
much more than the information taught by the current MTIP. 
Expertise can now be gained only through years of job 
experience, on-job-training (OJT) and study. There is 
potential for an Expert System to capture an aggregate of 
existing expertise for various ratings. Using Expert System 
training, expertise may be transferred to students in days 


and months instead of years. 
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Furthermore, an Expert System may raise 
individualized training, a key concept of the MTIP, to a new 
level. It can provide unique initial, corrective and 
remedial training for each student based on the student 
profile. It can "intelligently" determine trainee 
comprehension, generate problems for the student and 
determine the method of presentation. Each trainee receives 
instruction designed specifically for him. Another feature 
of an "intelligent" system is that it can respond to a much 
wider range of communication with humans. It can make a 
system much more "user friendly." The U. S. Navy has 
sponsored research to develop an "intelligent" interface to 
CBT [Bryant 88:pp. 199-206]. Future developments may have 
application for the MTIP. 

CBT offers the possibility of considerably expanding 
the scope of the current MTIP. The Air Force Human 
Resources Laboratory (AFHRL) has made an initial effort to 
develop an Intelligent Maintenance Advisor. The envisioned 
role of the Advisor is to (1) enable unskilled technicians 
to perform as if they were skilled ("skill multiplier"), (2) 
tutor technicians who need further development of their 
technical skills and (3) work cooperatively with skilled 
technicians to capture diagnostic insights (expertise) for 
the benefit of their successors. Such a device retains its 
training function and also serves as a "skill integrator" to 


interactively assist the experienced technician with 
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possible solutions to a problem being considered. Finally, 
mms technology could provide the MTIP with an authoring 
System to create new training knowledge base or modify 
existing knowledge bases. [Richardson 86:pp. 47-48] 

4. Instructional Systems Development 

_Macomnon methodology Lorsconducting CBT is 
Instructional Systems Development (ISD). The Commander, 
wat Air Force, United States Pacific Fleet, has directed 
that test questions be generated for MTIP using ISD [CNAP 
foep. |. The most important goal of ISDwis to provide 
performance oriented training. 

CBT is the most feasible method to support an 
extensive ISD framework for a large scale training program 
like MTIP. The performance oriented philosophy differs from 
miemthaditional curriculum approach. ISD focus on providing 
consistent training to improve future job performance while 
MiemtrGaditional approach centers on the training of topics 
developed individually each time the training is offered, 
without regard to how the training will be used. Table 5.1 
illustrates the difference between MTIP using the ISD and 
traditional approaches to instruction. [Kearsley 84:pp. 83- 
84 | 

AMOtHemRY Characteristic Ofb@lSD applicable tomMTIP, is 
that empirical evaluation is an integral part of the 
training program. In the course of training, the user and 


the managers may detect deficiencies in the program. Also, 
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the needs of the user or the Navy may change with time. A 
computer-based training program may be altered and 
customized to meet the needs of the Navy or users. 
Corrections may be implemented promptly. Service-wide 
alterations may be incorporated rapidly with good 
configuration control and high confidence of standardization 
where needed. 

A third characteristic of ISD is the attempt to 
define optimal strategies, media and sequencing for selected 
training activities. While MTIP does seek to find the 
optimal strategy to conduct training, the use of limited 
media and inability to optimally control sequencing also 
apply. CBT can coordinate the availability of many media-- 
print, slides, video, computers, simulators, embedded 
training and expert systems--to illustrate a range of 
possibilities. [Kearsley 84:p. 83] 

ISD may be applied to both computerized and non- 
computerized training. MTIP has employed the most basic 
features of CBT. Computerized impact has primarily been 
along the lines of Computer Managed Instruction (CMI) to 
meet administrative needs. Future developments and 
enhancements for MTIP will likely involve more application 
of CBT and are discussed later. The large size of MTIP, 
potential high cost for further evaluation and maturation, 
and the commitment to ISD suggest a phased approach with 


increased CBT for MTIP. 


oo 


ABLE 5.1 


MTIP using ISD 


Content based upon 
performance 


Instructional objectives are 
explicitly stated 


Content is developed by 
multidisciplinary team 


Evaluation is part of 
development process 


Media, strategies and 
sequencing are analytically 
derived 


Materials take into account 
student capabilities and 


differences 


Development process and 


implementation are consistent 


and reliable across 
individuals 


COMPARISON OF MTIP WITH ISD AND TRADITIONAL 
APPROACHES TO INSTRUCTION 


[Kearsley 84:p. 84] 


Traditional 


Content based upon topics to 
covered 


Instructional objectives are 
aire kane ets 


Content is developed 
pe mara lye ysis ere -t Or 


EvValivliat@ron occurs atter 
training is implemented 


Media, strategies and 
sequencing are intuitively 
selected 


Materials are designed for 
ideal or average student 


Development process and 
implementation are 
idiosyncratic and vary with 
instructor 


ISD has been most widely and thoroughly applied in 


the military training environment. 


This is because of the 


long lead time and expense required for the most effective 


training program and the severe consequences of inadequate 


training programs. 


Table 5.2 depicts the benefits to MTIP 


of the ISD methodology compared to traditionally developed 


instruction. [Kearsley 84:pp. 
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TABLE 5.2 BENEFITS FOR MEPEP FROM Secor AR ED Lo 
TRADITIONALLY DEVELOPED INSTRUCTION 
[Kearsley 84:p.88] 


Training closely matches job requirements and 
student needs 


Training takes significantly less time to complete 


Training is highly consistent across different 
occasions and instructors 


Training can meet a wide range of variation in 
Student entry levels and abilities 


Improved confidence, morale and satisfaction on the 
part of students 





ISD has been described as consisting of five major 
phases which are illustrated in Figure 5.1. Each phase may 
include several steps depending upon the application. 
[Kearsley 84:pp. 85-86, Logan 82:pp. 6-12] These phases are 
relevant to MTIP development and enhancement. 

a. Analysis 

In the Analysis phase, needs are assessed with 
respect to a performance problem or a new requirement. The 
nature and scope of the training required are identified. 

Is MTIP training the appropriate solution? Perhaps a 
personnel, organizational, procedural or job aid solution is 
needed. Also, available acum and the time and schedule 


constraints for a prospective MTIP must be identified. 
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Additionally, the jobs or tasks to be taught must be 
identified by the functions to be performed. Finally, 
students to be trained should be analyzed to determine 
aptitudes, motivation, reading ability and prior job 
knowledge and experience. Can individual differences be 
accommodated by MTIP training? 
b. Design 
The Design phase is concerned with defining 
instructional objectives, specifying lessons and tests and 
selecting the applicable media. Instructional strategies 
and sequencing based on aircraft systems are determined. [In 
this phase the nature of the training can be specified. 
c. Development 
Lessons and tests are written based on input 
from aircraft maintenance experts. This involves technical 
writing and the production of manuals, slides, video or CBI. 
Also, prepared are instructor guides and materials for 
administration of MTIP such as training plans, evaluation 
forms and student records. 
d. Implementation 
Implementation is the delivery and management of 
training. Instructors and staff must be trained to use CBT 
such as the MTIP. Facilities are readied while time and 
space are allocated for training. Training is conducted, 


students learn, take tests and receive remedial training as 
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necessary. Records are kept and data is collected, 
distributed and stored. 

e. Evaluation 

As shown in Figure 5.1, Evaluation occurs 

throughout all phases of ISD. Evaluation provides feedback, 
control and regulation for the entire process. CBT can 
provide this capability. Initially, evaluation is made to 
improve instruction (formative). Later, judgments are made 
about the effectiveness of training (summation). As 
Evaluation controls the ISD process, it may also trigger the 
beginning of a new process or the termination of ISD for a 
particular environment. [Kearsley 84:pp. 85-87, Logan 82:pp. 


6-13] 


fee SUMMARY 

This chapter portrayed application of Computer-Based 
Training (CBT) to both the present and future Maintenance 
Training Improvement Program (MTIP). The present MTIP was 
described briefly. Several proposals were made to improve 
the current MTIP through better database management 
techniques. Progressive database management provides a much 
more responsive system and one that is more in tune with the 
needs of user operators. 

Decision support available in a CBT system, could 
provide MTIP managers and users with much more suitable 


information with which to manage. With this support, MTIP 
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resources could be better used to meet training needs. With 
efficient resource use and better management information 
readily available, MTIP effectiveness will certainly improve. 

Expert Systems were described as a means of significantly 
altering the training process itself. It may become 
possible to teach experience as well as information. Expert 
Systems may permit more integration of actual maintenance 
and the training to do maintenance. CBT has great potential 
to provide enhancements to the current MTIP and promote 
progressive future developments. 

Finally, Instructional Systems Development (ISD) was 
presented as the preferred method of MTIP with CBT 
implementation. In fact, current policy directs that test 
questions for current MTIP applications be generated with 
ISD. Some of the characteristics of ISD are described and 
the five major phases are illustrated. ISD was depicted as 
being performance oriented and several of the major benefits 


for MTIP are portrayed. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


The installation of computers into the classroom alone 
does not result in Computer-Based Training (CBT). Rather, 
CBT draws from both educational and computer technology to 
add new dimension to training. CBT has been implemented for 
a number of reasons. These reasons may generally be 
categorized as those which reduce training costs and/or 
those which improve training effectiveness. CBT has not 
been a panacea for all training environments but has worked 
well in many military training centers. 

The Maintenance Training Improvement Program (MTIP) is 
already a computer-based training program which can benefit 
substantially from increased employment of CBT! MTIP can be 
impacted by CBT in the two areas of database processing and 


management and, secondly, decision systems. 


A. DATABASE PROCESSING AND MANAGEMENT 

Database processing is an important characteristic for 
Gone A typical military training program has a formidable 
record-keeping requirement which may not be adequately 
recognized. Modern database processing streamlines record- 
keeping and facilitates training and training management by 
providing the most relevant data and information as needed. 


Furthermore, a sound Database Management System (DBMS) 
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establishes a foundation for additional computer-based 
enhancements for CBT. 

The MTIP needs Database Managers who have a sense of 
ownership for their databases and maintain empathy with the 
users. This attitude will result in a training program 
which is robust and responsive to operator inputs and needs. 
Databases must be rapidly up-dated as change recommendations 
are approved or new data and procedures become available. 
MTIP databases should serve as libraries for their 


respective type of aircraft. 


Bs “DECI SVON Soars eats 

CBT can incorporate features of a Decision Support 
System (DSS). Managers of CBT are required to make 
decisions regarding CBT and students. Many decisions are 
routine and well-structured. A DSS; however, lends aid for 
those training management decisions which are unstructured. 
All levels of management and many types of decisions are 
supported by DSS leading to a more conducive training 
environment for students enrolled in CBT. 

MTIP managers need routine structured reports to control 
the daily operations of the program. A DSS capability would 
also provide support for resource optimization concerns and 
future oriented considerations. DSS provide mathematical, 
operations research and statistical tools to aid managers 


with these types of unstructured decisions. With this kind 
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of support, managers can maintain a state-of-the-art system 
by optimizing available resources and choosing the needed 
hardware, software and training technology which will bring 
maximum future benefit. 

Expert Systems have found limited application to date in 
CBT. Their potential, however, is very significant. Expert 
Systems offer the possibility of teaching another dimension 
of knowledge, namely, heuristics. These systems can help 
students learn in a classroom from the experience and 
expertise of others. This can provide students with a level 
of knowledge which has traditionally not been taught by 
conventional methods. 

The MTIP is an excellent candidate for Expert System- 
based training. It does not concentrate on topical 
classroom training; rather, it is action and performance 
oriented. The MTIP does teach basic facts and procedures, 
but its major focus is to train students to think and act 
like successful, experienced aircraft maintenance 


technicians. 


C. MAINTENANCE TRAINING IMPROVEMENT PROGRAM MANAGEMENT 

It is apparent that equipment and systems to be 
maintained by military personnel are becoming more 
sophisticated and technically advanced. CBT offers the 
potential to bring substantial computing power to personnel 


training. For CBT to be effective it must be regarded as a 
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new technology. It is not merely a combination of existing 
educational and computer technologies. The adequacy and 
quality of CBT depend on the effort which goes into its 
planning, design, development and implementation. The 
various aspects of computing power--database processing and 
Management, decision support and Expert Systems--bring a 
synergy to CBT which can dramatically improve the training 
provided. 

The primary thrust is that CBT may be more fully applied 
to the Maintenance Training Improvement Program (MTIP). 
Both the current MTIP arrangement and future developments 
and enhancements can benefit greatly from a systematic CBT 
approach. Considerable work remains to develop CBT 
technology for MTIP to the level described here. Decision 
systems for training are in their infancy. Personnel 
policies must ensure the adequate recruitment and assignment 
of people to do MTIP development, implementation and 
maintenance. Sites must be selected to test prototype 
developments. 

Thus, naval aviation maintenance technicians can be 
better prepared to meet the challenges of an increasingly 
complex environment where high, sustained equipment and 


system readiness is expected and required. 
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riage tN DIX A 
DE Neon OF st ERM 


There is a proliferation of terminology associated with 
CBT. The field is made up of people and commercial vendors 
with very diverse backgrounds. CBT also has a wide variety 
of applications and considerable variability in the uSe of 
terms is found in the literature. Some of the most common 
terms are listed to explain their meaning and the concepts 


which the terms represent [Burke 82:pp. 180-181]. 
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DEFINITION OF TERMS 


Automated Instructions 4 Any type of instruction in 
which administration or delivery is controlled by a 
computer. 


Automated Teaching (AT) See Automated Instruction. 
Automated Training (AT) See Automated Instruction. 
Computer-Aided (Assisted) Instruction (CAI) An 


application of Computer-Based Training that involves an 
on-line interactive learning process where a computer 
is the primary delivery systen. 


Computer-Aided (Assisted) Education (CAE) See Computer- 
Aided Instruction. 


Computer-Aided (Assisted) Learning (CAL) See Computer- 
Aided Instruction. 


Computer-Aided (Assisted) Training (CAT) See Computer- 
Aided Instruction. 


Computer-Based Education (CBE) See Computer-Based 
Lrainine- 


Computer-Based Instruction (CBI) See Computer-Based 
ie aan in or 


Computer-Based Training (CBT) An interactive computer 
training system in which a computer is the primary 
delivery system and is used to provide overall training 
management. 


Computer-—Managed Education (CME) See Computer-Managed 
Instruction. 


Computer—Managed Instruction (CMI) An application of 
Computer-Based Training which uses a computer to direct 
students through their training and produces reports on 
student performance or system utilization. It has 
elements in common with Computer-Aided Instruction, but 
Computer-Managed Instruction usually refers to a stand- 
alone system. 


Computer-Managed Learning (CML) See Computer-Managed 
InStruction, 
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Computer-Managed Training (CMT) see Computer-Managed 
limstrEuction . 


Conventional Instruction (CI) Classroom, laboratory or 
lecture instruction which is presented to a group of 
Students as a whole. Students usually have similar 
academic aptitudes and all receive the same 
instruction. Instruction is paced at a rate to 
accommodate all or nearly all students. 


Embedded Computer-based Training Interactive training 
which is an integral part of equipment or a system. 


iiervidualized Instruction (II) Instruction which 
accommodates individual differences in aptitude, skill 
level, prior knowledge and learning style. It is 
characterized by minimal time constraints, choice of 
instructional medium and adjustments to the training 
program as it progresses based on feedback. 


Intelligent Computer-Aided Instruction (ICAI) A 
Computer-Aided Instruction (see) system implemented by 
Expert System knowledge base techniques. 


Programmed Instruction (PI) hrstiewe rk womein wien 
material is presented cumulatively in small steps and 
progresses as the student masters each step. It may be 
presented by either computerized or noncomputerized 
media. 


Self-Paced Instruction (SPI) [instruction which allows 


students of different aptitudes and prior knowledge to 
progress through a training program at their own rate. 


Los 


AP EE Nee 
ACRONYMS 


There are many acronyms associated with Computer-Based 
Training (CBT) and computer technology. A military 
application may introduce unfamiliar acronyms for activities 
as well as systems or concepts. A reference source is 


provided to specify and clarify acronyms applicable to 


NplrtarveceL. 
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AFHRL 


AIS 
CAI 
CBT 
CCP 


CES 


CNET 


CMI 
DBMS 
DSS 
EDP 


FRAMP 


ICAT 


IDA 


iD 
KBMS 
MBMS 
MIS 
MTP 
N AMP 


NATTC 


ACRONYMS 
Air Force Human Resources Laboratory, 
Brooks Air Force Base, Texas 
Advanced Instructional System 
Computer-Aided Instruction 
Computer-Based Training 
Communication Control Program 


Contractor Engineering Technical 
services 


Chief of Naval Education and Training, 
Pensacola, Florida 


Computer-Managed Instruction 
Database Management System 
Decision Support System 
Electronic Data Processing 


Fleet Readiness/Replacement Aviation 
Maintenance Personnel 


Intelligent Computer-Aided Instruction 


Institute for Defense Analysis, 
Arlington, Virginia 


Instructional Systems Development 
Knowledge Base Management System 

Model Base Management System 

Management Information System 
Maintenance Training Improvement Program 
Naval Aviation Maintenance Program 


Naval Air Technical Training Center, 
Millington, Tennessee 


EOS 


NAVAIR 


Nee 


NPRDC 


OTE 


PLATO 


2DS 
OME 


LIiCewe! 


Naval Air Systems Command, Washington, 
eer 


Navy Engineering Technical Services 


Navy Personnel Research and Development 
Center, San Diego, California 


On-Job-Training 


Program Logic for Automatic Teaching 
Operations 


Structured Decision System 
Subject Matter Expert 


Time Shared Interactive Computer- 
Controlled Information Television 
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APPENDIX C 


LIST OF RELATED WORKS 


The Related Works are categorized as being most 


applicable to one or more of the following topics: 


Computer-Based Training and Education 
Database Processing and Management 
Decision Support 


Expert System and Knowledge Technology 


Or 


List OF VEELATEDS WORKS 


Computer-Based Training and Education 


Air Force Human Resources Laboratory (AFHRL), Brooks AFB, 


TX, TP-82-24, Computer Based Maintenance Aids System; 


Preliminary Development and Kvaluaction of Ja Proeouype mum 
Donald L. Thomas, September 1982. 


Air Force Human Resources Laboratory (AFHRL), Brooks AFB, 


TX, TR-70-24, Application of Computers in Educational and 
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Instructional Centers, by Gerald S. Walker and Edward M. 


Gardner, December 1970. 
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Ash, Ruth Cranford, An Analysis of the Cost-Effectiveness of 
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Dissertation, Auburn University, Auburn, AL, 1985. 
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v. 12, pp. 149-162, February 1987. 
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v. 162(3857), pp. 970-976, 29 November 1968. 


Science, 


Buck, John A., "Federal Aviation Administration Seventh Youn 
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Instruction, v. 8C3), pp. 53-55, febriaryv eee 


Chief of Naval Education and Training, Pensacola, FL, TAEG 
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Chief of Naval Education and Training, Pensacola, FL, TAEG 


Technical Report No. 44, Computer Managed Instruction at 
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December 1976. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


Technical Report No. 49, Computer Managed Instruction at 


Remote Sites: Phases II-III, A Demonstration Design, by 
Kenneth A. Polcyn, E. Scott Baudhuin, Lawrence T. Brekka, 


Margaret H. Hoyt, Bernard E. Keiser, Della M. Kennelly and 
Bernard H. Rudwick, December 1977. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 
Technical Report No. 78, An Assessment of Individualized 
ieteuction in Navy Technical Training, by M. Michael 
Zajkowski, Edward A. Heidt, James M. Corey, Dorothy V. Mew 
and Gene S. Micheli, November 1979. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


Technical Report No. 89, Computer Based Instructional 


pyestems...1985 to 1995, by Gene S. Micheli, Charles L. 
Morris and William M. Swope, August 1980. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


Technical Report 117, A Comparison of Individualized & 


Somventional Instruction in Navy Technical Training, by 
Eugene R. Hall and Jon S. Freda, March 1982. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


Secunical Report 119, Analysis of Factors Affecting the 


Performance of the Navy's Computer Managed Instructional 
eyocem, by William M. Swope, James M. Corey, Richard M. 


Evans and Charles L. Morris, April 1982. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


meemnucal Report 141, The Application of Distributed System 
Mmeemmbectunres for Computer—Based Instructional Systems in 


the Naval Education and Training Command, by Barney L. 
Capehart and Charles L. Morris, Jr., February 1983. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 
Technical Report 147, Self-Paced and Conventional 


Instruction in Navy Training: A Comparison on Elements of 
Quality, by Richard M. Evans and Richard Braby, July 1983. 


Chief of Naval Education and Training, Pensacola, FL, TAEG 


Technical Report 152, Design, Development, Implementation, 
and Evaluation of the Computer Aided Interactive Testing 
System (CAITS), August 1983. 


109 


Chorover, Stephan L., “Cautions on Computers in Hamicationmee 
BYTE, vs. 9, pDpwj 223-276 nine ceca 


Coffee, Michael B., "Computerized Simulators Offer Quality 
Training at Modest Cost," Power, edited by Tom Elliott, 
v. I31, Dd. 55-58, Jantianry sl or. 


Comeau, B. J. (Perry), "Does Your CBT Course Teach 


Performance?" Training and Development Journal, v. 40(7), 
July 1986, pp. 42-44, 


Eller, Ben F., Kaufman, Alan S., and McLean, James E., 
"Computer-Based Assessment of Cognitive Abilities: Current 


Status/Future Directions," Journal of Educational Technology 
systems, vs 1502). pp. 137-147 lose er. 


Ellis, John A., editor, Malitan Contr i buteoncmed 
Instructional Technology, Praeger Publishers, 1986. 


Federico, Pat-Anthony, “Individual Differences in Cognitive 
Characteristics and Computer-Managed Mastery Learning," 


Journal of Computer-Based Instruction, v. 9(1), pp. 10-18, 
Summer 1982. 


Federico, Pat-Anthony, "Changes in the Cognitive Components 
of Achievement as Students Proceed Through Computer-Managed 


Instruction," Journal of Computer-Based Instruction, v. 
9(4), pp. 156-168, Spring 1983. 


Furr, Coleman, The Effect of Ungraded Testing on Student 
Learning in Computer Managed Instruction, Ph.D. 


Dissertation, United States International University, San 
Diego, Caw loeze 


Gery, Gloria, Making CBT Happen, Weingarten Publications, 
bic. oe. 


Gray, Susan H., "The Effect of Sequence Control on Computer 


Assisted Learning," Journal of Computer-Based Instruction, 
v« 402), pp. 54-56). opr mse 


Halff, Henry M., Hollan, James D., and Hutchins, Edwin L., 
"Cognitive Science and Military Training," American 
Psychologist, v. 41(10), pp. 1131-11 3o Uc eante ee sce 


Hassett, James and Dukes, Sheree, "The New Employee Trainer: 


A Floppy Disk," Psychology Today, v. 20(9), pp. 30-36, 
September 1986. 


Henderson, Peter B., "A Computer-Aided Instructional 


Environment," Journal of Educational Technology Systems, v. 
15(¢3), pp. 281-301, 1986-87. 


Hendrickson, C., Pasquale, A., Robinson, W., and Rossi- 
Velasco, M., "Applications of Computer Aided Instruction," 


Journal of Professional Issues in Engineering, v. 112, pp. 
194-202, July 1986. 


Hillelsohn, Michael J., “How to Think About CBT," Training 


and Development Journal, v. 38(12), pp. 42-44, December 
1984. 


Institute for Defense Analysis (IDA), Arlington, VA, Paper 
P-1568, Cost-Effectiveness of Maintenance Simulators for 
Peeeatary Iraining, by J. Orlansky and J. String, August 
resi. 


Kearsley, Greg P., and Hillelsohn, Michael J., "Human 
Factors Considerations for Computer-Based Training," Journal 


of Computer-Based Instruction, v. 8(4), pp. 74-84, May 1982. 


Kedem, Avi and Smilansky, Jonathan, "Maintenance Training 


Packages: An Integrated Approach," Training and Development 
Journal, v. 40(6), pp. 62-65, June 1986. 


Kontos, George, “Instructional Computing: In Search of 
Better Methods for the Production of CAI Lessons," Journal 


of Educational Technology Systems, v. 13(1), pp. 3-14, 1984- 
S5 . 


Korbak, Michael, Jr., Computer-Managed Instruction: f[heory, 


Application, and Some Key Implementation Issues, Masters 
Thesis, Naval Postgraduate School, Monterey, CA, March 1984. 


Lahey, George F., "The Effect of Instructional Sequence on 
Performance in Computer-Based Instruction," Journal of 


Computer-Based Instruction, v. 7(4), pp. 111-116, May 1981. 


Lindahl, Charles E., "Applying Computer Technology to 


Training Systems," Defense Management Journal, v. 16(4), 
pp. 38-43, Fourth Quarter 1980. 


Medsker, Karen L., and Medsker, Larry R., “Instructional 
Technology: A Key to Successful Information Systems," 


Information & Management, v. 12(4), pp. 195-208, April 1987. 


Meeks, Brock N., “The Quiet Revolution," BYTE, v. 12, 
pp. 183-190, February 1987. 


Naval Training Systems Center~Orlandc, ER kee, eu. 
Individual Differences in Military Training Environments: 
Four Areas of Research, by Joyce Hogan, Steven Arneson and 
Eduardo Salas, April 1987. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, SR 83-13, Development of a Computer-Based 
Tactical Training System, by Alice M. Crawford and James D. 
Hollan, January 1983. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, SR 83-43, Evaluation of the GCMiyinstusucron 


Role Training Program in the Navy and Air Force, by Barbara 
L. McCombs and Jacqueline L. Dobrovolny, June 1983. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TN 87-4, Computer-Based System for Assessing 


semantic Knowledge: Enhancements, by Nina L. Liggett and 
Pat-Anthony Federico, December 1986. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TN 87-25, Characteristics of Navy Training 


Courses and Potential for Computer Support, by C. Douglas 
Wetzel, Donald L. Van Kekerix and Wallace H. Wulfeck II, 


May 1987. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TR 82-16, Computer-Managed Instruction in the 
Navy: V. The Effects of Charted Beedbachwone.aecmoe 


Progress Through a CMI Course, by N. H. Van Matre, H. S&S. 
Pennypacker, W. H. Hartman, Ba ES Brett sand EO. ward. 


November 1981. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TR 82-19, Computer-Managed Instruction in Navy 
Technical Training: An Attitudinal Survey, by Carol Ann 


Robinson, Elizabeth A. Tomblin and Archester Houston, 
December 1981. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 82-53, An Evaluation of Individualized, Job- 


Specific Maintenance Training, by Kirk A. Johnson, Linda L. 
Graham and Stuart B. Carson, July 1982. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 83-31, The Instructional Quality Inventor 
(1QI): Evaluation Tool for Instructionat@eoysrens 
Development, by William E. Montague, John A. Ellis and 
Wallace H. Wulfeck II, August 1983. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TR 84-29, Computer-Based Instruction: Stability 
of Cognitive Components, by Pat-Anthony Federico, February 
1984. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 84-30, Computer-Managed Instruction: 
Individual Differences in Student Performance, by Pat- 
Anthony Federico, February 1984. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TR 85-11, Computer-Based Systems for Navy 
Classroom Training, by William E. Montague, November 1984. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 87-14, Cognitive Factors in Learning and 
Retention of Procedural Tasks, by Paula J. Konoske and John 
A. Ellis, December 1986. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 88-2, Field Evaluation of Simulation for 
Maintenance Training: SH-3 Helicopter Electro-Mechanical 
ieaa@efold System, by V. M. Malec and M. Luszczak, October 
oS 7 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TR 88-3, Analysis of Navy Technical School 


iraining Objectives for Microcomputer Based Training 
systems, by C. Douglas Wetzel, Donald L. Van Kekerix and 


Wallace H. Wulfeck II, October 1987. 


O'Neil, Harold F. Jr., editor, Computer-Based Instruction, 


Academic Press, Inc., 198l. 


Orlansky, Jesse and String, Joseph, "Computer-Based 
Instruction for Military Training," Defense Management 
Journal, v. 17(2), pp. 46-54, Second Quarter 1981. 


Pressman, Israel and Rosenbloom, Bruce, "CAI: Its Cost and 


Its Role," Journal of Educational Technology Systems, v. 
m3). pp. 183-208, 1983-84. 


Rockart, John Fralick and Scott Morton, Michael S., 


Computers and the Learning Process in Higher Education, 
McGraw-Hill Book Company, 1975. 


Sassone, Peter G., "The Economic Evaluation of Military 


Training Programs," Defense Management Journal, v. 18(4), 
pp. 09-42, Fourth Quarter 1982. 


Steinberg, Esther R., Teaching Computers to Teach, Lawrence 


Erlbaum Associates, Inc., 1984. 


Database Processing and Management 


Date, C. J., An Introduction to Database Systems, Volume II, 
Addison-Wesley Publishing Company, 1983. 


Katzan, Harry, Jr., Computer Data Management and Data Base 
Technology, Van Nostrand Reinhold Company, 1975. 


House, William C., editor, Interactive Decision Oriented 
Data Base Systems, Mason/Charter Publishers, Inc., 1977. 


scott Morton, Michael 5., Management Decision Systems, 


maevard University, 1971. 


Decision Support 


Ginzberg, Michael J., Reitman, Walter and Stohr, Edward A., 
editors, Decision Support Systems, Proceedings of the NYU 
Symposium on Decision Support Systems, 21-22 May 1981, 
North-Holland Publishing Company, 1982. 


House, William C., editor, Interactive Decision Oriented 
Data Base Systems, Mason/Charter Publishers, Inc., 1977. 


Hughes, Cary T., "Relationships Between Demographics, 
Training, etc. in a DSS Environment," Information & 


Management, v. 12(5), pp. 257-261, May 1987. 


Mittra, Sitansu S., Decision Support Systems, John Wiley & 
Sons > ‘ine. 2 hoe. 


Navy Personnel Research and Development Center (NPRDC), San 


Diego, CA, TN 87-4, Computer-Based System for Assessing 


semantic Knowledge: Enhancements, by Nina L. Liggett and 
Pat-Anthony Federico, December 1986. 


Navy Personnel Research and Development Center (NPRDC), San 
Diego, CA, TR 87-14, Cognitive Factors in Learning and 


Retention of Procedural Tasks, by Paula J. Konoske and John 
A. Ellis, December 1986. 


Sprague, Ralph H., Jr., and Watson, Hugh). editors. 
Decision Support Systems, Prentice-Hall, 1986. 


Scott Morton, Michael S., Management Decision Systems, 
Harvard University, 1971. 


seidel, Robert J., and Weddle, Peter W., editors, Computer 


Based Instruction in Military Environment, Plenum Press, 
1987. 


Pie 


Expert System and Knowledge Technology 


"Artificially-Intelligent Maintenance Diagnostics," 
pammlation, v. 48(2), pp. 74-75, February 1987. 


Brodie, Michael L. and Mylopoulos, John, editors, On 


Knowledge Base Management Systems, Springer-Verlag, Inc., 
1986. 


Clancey, William J., Knowledge-Based Tutoring: The GUIDON 
Program, The MIT Press, 1987. 


Harris, Steven D., and Owens, Jerry M., "Some Critical 
Factors That Limit the Effectiveness of Machine Intelligence 
Technology in Military Systems Applications," Journal of 


Computer-Based Instruction, v. 13(2), pp. 30-34, Spring 
1986. 


itetitute of Cognitive Science, University of Colorado, 
Boulder, CO, Technical Report No. 142, A Multimedia 


Knowledge Representation for an "Intelligent" Computerized 
Tutor, by Patricia Baggett and Andrzej Ehrenfeucht, April 


Pipitone, Frank, "An Expert System for Electronics 
Troubleshooting Based on Qualitative Causal Reasoning,” 


Journal of Computer-Based Instruction, v. 13(2), Spring 
1986, pp. 39-42. 


Seidel, Robert J., and Weddle, Peter W., editors, Computer- 


Based Instruction in Military Environment, Plenum Press, 
1987. 


eeeeman, D., and Brown, J. S., editors, Intelligent Tutoring 
Systems, Academic Press, Inc., 1982. 


U. S. Army Research Institute for the Behavioral and Social 
Sciences, Alexandria, VA, Research Report 1412, Smart 
Technology for Training: Promise and Current Status, by 
Wayne D. Gray, Daniel B. Pliske and Joseph Psotka, March 
eS . 


University of Pittsburgh Technical Report No. LSP-3, 


Pittsburgh, PA, An Object-Oriented Architecture for 


Intelligent Tutoring Systems, by Jeffrey Bonar, Robert 
Cunningham and Jamie Schulz, 14 August 1987. 


REFERENCES 


[Baker 81] Baker, Frank, "Computer-Managed 
Instruction: A Context for Computer-Based 


Instruction," Computer-Based Instruction, edited by 
Harold F. O'Neil, Jr., Academic Press, Ines. 106i 


[Bevan 87] Bevan, S. J., and Wetherall, P. Reed 
Conversion of an Expert System to an Intelligent CAI 


System," Computer-Based Instruction in Military 
Environments, edited by Robert J. Seidel and Peter D. 


Weddle, Plenum Press, 1987. 


[Bork 87] Bork, Alfred. Ihe Potential tom 
Interactive Technology," BYTE, v. 12, February 1987. 


[Bryant 88] Bryant, Robert A., "An ‘Intelligent’ 


Interface to CBT," Journal of Educational Technology 
Systems, v. 16(3), 1987-88. 


[Burke 82] Burke, Robert L., CAI Sourcebook, 
Prentice-Hall, 1982. 


[Cesena 87] Cesena, Mary L., and Jones, Wendell O., 
"Accelerated Information Systems Development in the 
Army," Spectrum, v. 4(3), September 1987. 


[Clancey 87] Clancey, William J., Knowledge-Based 
Tutoring: IJVhesGUEDON Program, Phe Mil (ress, vere 


[CNAP 86] Commander, Naval Air Force, United 
States Pacific Fleet (COMNAVAIRPAC), San Diego, CA, 
Instruction.1543.1, AprYrewle ee eoece 


[Date 86] Date, C. Js, An Int roductionmeo 
Database Systems, Fourth edition, Volume I, Addison- 
Wesley Publishing Company, 1986. 


[Davis 83] Davis, William S., Systems Analysis and 
Design, Addison-Wesley Publishing Company, 1983. 


[Dean 83] Dean, Christopher and Whitlock, Quentin, 


A Handbook of Computer-Based Training, Nichols 
Publishing Company, 1983. 


[Fletcher 86] Fletcher, J. Dexter and Rockway, Marty 
R., "Computer-Based Training in the Military," 


(Mit abyecCOnenrowtlons to Instructional Technology, 
edited by John A. Ellis, Praeger Publishers, 1986. 


[Gorry 71] TOErry,uG. santhony and scott Morton, 
Michael S., "A Framework for Management Information 


Systems," Sloan Management Review, v. 13(1), Fall 1971. 


[Hansen 75] Hansen, Duncan N., Ross, Steven M., 
Bowman, Harry L., and Thurmond, Paul, Navy Computer 
Managed Instruction: Past, Present, and Future, 
Memphis State University, Memphis, TN, Bureau of 
Educational Research and Services, 1975. 


(Harmon 85] Harmon, Paul and King, David, Expert 
oxpere 
Systems, John Wiley & Sons, Inc., 1985. 
[Head 67] Head, Robert V., "Management Information 
Systems: A Critical Appraisal," Datamation, v. 13(5), 
May 1967. 
[IDA P-1375, 79] Institute for Defense Analysis (IDA), 


Arlington, VA, Paper P-1375, Cost-Effectiveness of 


Computer-Based Instruction in Military Training, by 
Jesse Orlansky and Joseph String, April 1979. 


[Kearsley 83] Kearsley, Greg, Computer-Based Training, 
Addison-Wesley Publishing Company, Inc., 1983. 


[Kearsley 84] Kearsley, Greg, Training and Technology, 
Addison-Wesley Publishing Company, Inc., 1984. 


[Kearsley 85] Kearsley, Greg and Seidel, Robert J., 
Automation Aan training and Education,” Human Factors, 
vee 27 Cle 1985. 


[Kontos 84] Kontos, George, "Instructional 
Computing: In Search of Better Methods for the 
Emoduetton of CAl Lessons, Jounnal of Educational 


Technology Systems, v. 13(1), 1984-85. 


(Kroenke 83] Kroenke, David M., DATABASE PROCESSING: 
Fundamentals, Design, Implementation, Second edition, 


Science Research Associates, Inc., 1983. 
[Kroenke 87] Kroenke, David M., and Dolan, Kathleen 


A., Business Computer Systems, Third edition, 
Mitchell Publications, Inc., 1987. 


ie 


(Rogan. Logan, Robert S., Instructional Systems 
Development, Academic Press, Inc., 1982. 


[ Ly me ree Lynch, Robert Michael, A Cost 


Effectiveness Analysis of Computer Assisted Instruction, 
Ph.D. Dissertation, University of Northern Colorado, 
Greeley, CO, Winter 1971. 


[Mason 69] Mason, Richard 0., "Basic Concepts for 
Designing Management Information Systems," 1969, 


reprinted in Measurement for Management Decision, 
Addison-Wesley, 1981. 


[Mylopoulos 86] Mylopoulos, John, "On Knowledge Base 


Management Systems," On Knowledge Base Management 
Systems, edited by Michael L. Brodie and John 


Mylopoulos, Springer-Verlag, Inc., 1986. 


[NAMP 86] Chief of Naval Operations (OPNAV), 
Washington, D. €., Instruction 4790.2), October a 
1986. 

[NPRDC TN 82-17, 82] Navy Personnel Research and 


Development Center (NPRDC), San Diego, CA, TN 82-17, 


Computer-Based Education and Training Functions: A 
Summary, by R. D. Peters, May 1982. 


[NPRDC TR 82-45, 82] Navy Personnel Research and 
Development Center (NPRDC), San Diego, CA, TR 82-45, 
Instructor's Role in Individualized Training: A Survey 


of Two Computer-Managed Courses, by Kirk A. Johnson and 
Linda L. Graham, May 1982. 


[NPRDC TR 84-54, 84] Navy Personnel Research and 
Development Center (NPRDC), San Diego, CA, TR 84-54, 
Computer-Based Instruction: Will It Improve 


Instructional Quality?, by William E. Montague and 
Wallace H. Wulfeck II, August 1984, reprinted from 


Training Technology Today, v. 1(2), pp. 4-19, Winter 
1984. 


[O'Neil 81] O'Neil, Harold f.. Jr-- anderen is. 
Judith, "Introduction and Overview of Computer-Based 


Instruction," Computer-Based Instruction, edited by 
Harold F. O'Neil, Jr., Academic Press, 1981. 


(Parkinson 57] Parkinson, C. Northcote, Parkinson's 
Law, Houghton Mifflin Company, 1957. 


[Richardson 86] Richardson, J. Jeffrey and Jackson, 
Terresa E., "Developing the Technology for Intelligent 


Maintenance Advisors," Journal of Computer-Based 
Mistruction, Veml3(2Z),.) Springs 1986. 


ipeuek 86] RUC wiemariGak Wee sandukang, John W., 


"Training—Decision Systems," Military Contributions to 
g y seititary vontridutions to 


Instructional Technology, edited by John A. Ellis, 
Praeger Publishers, 1986. 


[Seidel 81] Seidel, Robert J., and Wagner, Harold, 


"Management," Computer-Based Instruction, edited by 
naeold £20 Neil seer. , Academic Press, 1981. 


fopiro 87 | Spiro, R. J., "“Experience-Consolidation 
Systems: A Sketch of a Theory of Computer-Based 
Instruction in 1I11-Structured Domains," Computer-Based 


Instruction in Military Environments, edited by Robert 
J. Seidel and Peter D. Weddle, Plenum Press, 1987. 


[Sprague 77] Sprague, Ralph H., Jr., and Watson, Hugh 
ie Mis Goneepts Part 1], Interactive Decision 


Oriented Data Base Systems, edited by William C. House, 
Petrocelli/Charter, 1977, reprinted from Journal of 


Systems Management, pp. 34-37, January 1975. 


[Sprague 80] Sprague, Ralph H., Jr., "A Framework for 
the Development of Decision Support Systems," MIS 
Quarterly, 4(4), June 1980, reprinted in Decision 


Support Systems, edited by Ralph H. Sprague, Jr., and 
Hugh J. Watson, Prentice-Hall, 1986. 


[Sprague 82] Wwheagic , whalpiemte, Jt a,. and Carlson, 


DEtew eye buaiding Eitfectave Decision Support Systems, 
Prentice-Hall, 1982. 


iMiarban $5] Turban, Efraim and Watkins, Paul R., 
"Integrating Expert Systems and Decision Support 
Systems," Transactions of the Fifth International 
foneerence on DSS (DSS "85)> reprinted in Decision 


Support Systems, edited by Ralph H. Sprague, Jr., and 
Hugh J. Watson, Prentice-Hall, 1986. 


[Wiederhold 86] Wiederhold, Gio, "Knowledge versus 


Data," On Knowledge Base Management Systems, edited by 
Michael L. Brodie and John Mylopoulos, Springer-Verlag, 


leer Ol 


id 


HO: 


INIT PEAS DISTRIBU PG Gas I 


Defense Technical Information Center 
Cameron Station 
Alexandria, Virginia 22304-6145 


Library,. Codes0142 
Naval Postgraduate School 
Monterey, California 93943-5002 


Director, Information Systems (O0P-945) 
Office of the Chief of Naval Operations 
Navy Department 

Wahingeton, D. C. 20350-2000 


Curricular Office, Codessor 
Computer Technology 

Naval Postgraduate School 
Monterey, California 93943 


Deputy Chief of Naval Operations (Air Warfare) 
Navy Department 
Washington, D. C. 20360-5100 


Commander, Naval Air Force 

U. S. Pacific Fleet 

Naval Air Station, North Island 
San Diego, California 92135 


Commander, Naval Air Force 
U. S. Atlantic Fleet 
Norfolk, Virginia 2351: 


Major John B. Isett 

Code 5418S 

Naval Postgraduate School 
Monterey, California 93943 


Lieutenant Colonel Gerald L. Pauler 
Code 55PA 

Naval Postgraduate School 

Monterey, California 93943 


Lieutenant Commander Galen J. Ledeboer 


USS Coral Sea (CV 43) 
FPO New York, New York 09550-2720 


122 





































































OF eee cement ars 
ere ere elena yf aplie4 
Pye et Te each el cen 
Seer PN el ie ie OT OO Ee atl Or CCR PE Py PT tee 
ee Pee Ot ee ae CY oe as 5 Te as eae Pd asa me Page 
rene eet 1 Sr re ale try ty of, oF 8! Mee eT tek pee Nr ey er Te ed oa 1% 
re LS eT eT yr  heeeiie POET teas per eT Lt alee rear eee Te ra | ~ 
Saar ere  Loets peer ee Ste rere Rg ay thesL3 
PaO aes oer tie 5B Slocd Mate S28 Oe ee ae bret Tot he s les 96 
TT" 2 peprrer ey Ltd ” Cr Oca te rare Le a PPe Le z 
meer ee fe ee §=6Computer-Based Tra lied 





ee oO Td Bl ater Tht) 
Oe a) aaa halons Prt PY Sons CT aoa Sas 
a 


EAT ewkartee cere PS TOL Lear; err rey Viet a ae oee ry, to 
Sera pee ee RC Rr rman | 
i Br ee de Pe ee ae A ppc | | | ] | | | | \ | | i | \\ | \y|I | | | | 
| II Hh | ) | I | 
WH | STAT \ iN 
} | Hii | | | y 
es : LINE | itl | | 










2 







rer) Pare oe ed eek 
red Of @ Ott The 





Soph Puet Lh chet 


ee Oe Cae 4 
i i 
Att D 





Te ee i adetht eer 
atahy te pry peer erry Tb had sletgrl 
Pd pe eh ts lndreth a are aT Pr | 
Pee ee Cr * , 
RT Ce ee Bad Pe 
WHY 
| 





oT neues 4 
y eve etd 
Pwo se | Perens 
pre pre gee TT) 
{ I 
Wil 


Phd 
Pl 















tJ 
POT ceed ane whp?.o-P ahs OF ea 
ST aon Oe ee att PT ee cect ae ak ae 
ge T TLD emt heer errr te he PY TL es Pee eee a ee: 
‘gh ovote Achig+¥ re peru ete we oN ee ] 
eT (TIT Te eh kas | 
ti | El i | ie 








td heeled 
Pert ee Te) a TTT le nett Mat 
Ae eget ed np Ol eels ye ee Prt 
pee eT ook cee alee Leela ee eT aebhds mere re veee Tr ee) 
I STE Reade Nd ae RPO ed ee 
4gto wie" lh ta a se ee a 


OT aaa 
el dd pear erie Te Ll adie Det Le od 
pepe el ea ate Oe te 
TT Le ed Ts 





are le 3 2768 000 84650 5 
DUDLEY KNOX LIBRARY 





























ny adele 
a OT = ering ti 
pe PPT tN or gr apon ent POS es ae Te a acai 
a wr ett Pe ea et Lad tet eo eel ” 
PET dae hohe een ern a he ag PP LE belies 
PT al eal - I Ie LL Le od ie Rata ¢ ad moi K' err 
WAT Oy ee dae ent A Be fp Ot Pee oe tne obneilek 
ere yt hb a La ohare haa ht Wray ers Te ed! 
oe oa) >» apf, @ at 
f Tree 


eT ae 

tT soematael ee 

nn egal Ny aabcte ideal gt Oe 

pr TT cade Te as Teh higtne te PT id ta al eT aaah 
at ae ee Oe TT Cree " s 

Per Le ee ad TTT LY ee ‘ tal eur 




































































































































































































































































































































































































































































































































































































































































































































eT all 
peaeey pe is ce Lael a hie este F metieah rr 
paper e eal paper tne Barn Te ar os 
pire I Sotetee etter ee eke ee a a ak par eer Pt ee 
ee aaele eee re ae as POE eo a aid Od OO eee od | Th ae 
ppteeginer Ore TT Liberte Or ta Gah & 41 ese.Gesetenas a kone Yr cry are) 
Pe el per eb e Reade rie ter Pree rvl ae eee 
TAP ie) 4 EF mA“ De hat at 90 aM 4 - Cue 
wr es he Pe oh allt f rer ay 
Cr ee ae 46 PN ea 
A eT ee are - 
eye oe arr a eed 
Pee eer aorawsd eere , ay 
Re eee] Pee Pf Me er oe 
erp rts ta yet! epee hg re Pe SL 
DE al ee WOOT Le, St 196 m! Pn ee 77 
er a dine pentired arati ia ee ehidel A SY cade . * Per rer et 
Seat etiaste Pi abeieged ae owes PPM Il baat eo Ne Se ; 
sale el dageaee ee alah arp eee Re POP ET ONE ala . i Ereerr Sep Ad odied 50% o . 
es PTTE ST atta dentaat eon I nT dircal tend PTY ti) cone pete eR PTT Deed neh tira } . ape ; No , : : a 
paagpee pat Pee alee wpe tre LI eer DP TY TT od, ‘ a a ae PR Do hare er he ee a : Fl at nse 
pasted Oo OT ocr pr aes » CTT a oat eel irr d OT Tr haber erty title, a - i ee , es «hay ra nieuw re ‘ ae = Py = 8 
pce OT hate ee Ue Le ee Pew eT Per cD Lemar eg Saher F sins PPR, * Nhe a CY a ad 4 " f «a = 
Oe edatngin ena fale ORT Ler parapets RP pre Onn a eal he APY re ee a Pere He eS - a ‘ 4 nde S 1 
a enable a Le per) TD a ees ae ap tlre Pn oe er rr tae Be are A 1 S , P Pf - A : . r - 
ey ad odeel l pe I Pet eae emssts moat gaae: Pee eee To rerpeye ty Pye SS pr Ot Bare , A ee ar e Py 
Py nie oor 6h et PT de dene Pre fod PT ileal ale Pe re, eer “ebgenarm ft yeaa aan 5 bh Oe ra 4 TT Ca ra ‘ Pi 
pe ae TT iad pee peeninnr een ete NOP ry nt at le ree TP lar art Sy + pe llega ei ¢ fi. * u u p 
wet nT lca Pree ge art err Ue ia ; pra ae Pre rae ae grote : Ue J wis f 7 ie ie Jee et nl 
fomhid "e Peeer ey Ct ny ee ieee ‘o" ere be Cee Act er Lee By sh ao? i ie ; , ' “+ a 
i ee oboe ea aut ; i , ese - Se ae 
ee re pba Pett P eto oe ; 7 iM rey Cae ve Y Me : | . eu a Fl 
3 ees =. Fee etl simieaten oe ; eet bL PA hs , r A rarer 1: , : 
eT fe nsee tans hag tat when see ee - Pat; Le ioe 4 ole a ie % » oer ek fy er f A A ' A ; : 
PPO rer Ladi) per reres Prete Lets : A ie % a Lets aes 4 - Ae 
wpe ye L peers er Ly Si - ; A dian 2 oe : 
—_ * © elated oF. “5 * eer ETAL, Sit Le : es ‘ 
ee th el ae al fue rp a Sah pe = eb oet 4 ; Ue ; Ue prof at s ea 
ony tia Oe abil he Pees a LM , M ct eB ig y : 
ST pant aaa pn ttn attend oe Ge enn a Pr ei A 5 Fs] Ae 
ee atime ere ere ate r Pe ae a be Sp . bau ’ FS = 
pe ee ert eT pp Te We SEU alae net r 3 A 
fon ee pr ers ap a TT oat POT ae eet f rare f RS od Fi 5 i 4 H A 
eT TT ako pt ee aha eter PUPAE eek ee A Aelia rr Tr | : et peda pe jaw ade : 9 AM le ee . i] es ' 
a cig Te OO rT anne S| A Nd bine seer deere SS VEY” ata Peep ree an Set hy eee ae S G 
cea Sphere RCT EE are Hi Pe Pe ea haL ah kage oligtats otin in ees . ey deel OE OP | aa J p a 4 8 p 
21 Vad 90R oe! a ch eee ne ee Te nee Per en Tn  t aPOrT ey gh Ls ; y 
Tr Aantal ahd a pt gsigt ommeegta® gle Pr 7 +n ee oes ee | Sedie te” fee "4 oa . r r] 
merpeecet ET ay Rarerant P-SOT TK Cr es oP » , : , 
aie foal imam? et L@amutiad 10 DP ot vat ere i ae 4 i ORT 26 
5 - ret r wie A 
Fe ee hal roy ror > a ° C = A 
" Pa ie Py Feat ne L L Pr ed 
Sy Pe eM " Le s a 
. r at ee es PP ae ‘ ; : 
raven coe. beaks atligns: rere eT rer ar a 4 ye 2» me TH 
ca ans 1 ofuprktace ge! CT te 1 rT py P| ee ae Pa del Sl 7 Le] » ' 
Seeere yr. F; y ripe arn ia PL Law ot es A = ; es 
= D ri = y*-' . M an LY ae a . 7 1 
Nel nephas PIP “ae Adee! ane ru 4 4 - a ™ at P 
pate Oe UO oF s uy Pe red Wet 7 
er de PS 7 Oe ASIC ; é = 2 ap ‘ 
ee TT . ‘ F 
ee a Led hd uy! : 6 Ce 
Pe ' b ' 8 r ® 7 
2 beg n One c be. ped eg Lt tas a “ . Ps af 28 F 
a eT 5 4 rs } a1 yi LF Ly x at] 1 ‘ 
. ; j J ihe a rps ect 1‘ ‘ . 
'. J ' n 
ey tit. 1 ae 
- ri , 
© re e a , Py 
A ~ 5 r 
eo J P iy 5 
1 t 
J ' a! 
; r f a. Pett re w ’ ° y 
a rl ere A P a oj * bs 
PY ree Un & Ae | p Fi ‘ i 
AF, rr) J bs a J ' 
i 2 ¢ br) 4 
5 eu ra n n F 4 
rr - ' : ‘ ‘| 
4 3 : oan ae : Bios E ; 
Py - if se be 1¢ r Par A 
s in H 
: a n , 1 At pO 5 4 y ' I 7 
Sod “3 : 4 
n ah 
P ' = ond x f° " s Pa A 
rd ar uy, rT 
i r bs 1 
Par P, y : 
f rie 2) P P 
¢ . 
. i F 5 sg 
hss 
‘ ' 2 a . . ; 
’ 1 J 
+ 3 US Joh , 
a t ; i 
5 m ad ' a 
~ , ? ' 
- rs 
° “Ve ; 
e res ord td Fe ad - 
elem ® rzht?." a woe A ax. : 
a Is ot dd nl 7 : I : 
Pd] ' temype' - 
5 titre. sit Ss oF ee x at 4 8 ; F 
Tht eta yout §eF- Pa PY x ! 
? ' i . 4 
a eee A >" , ; ip i) 
ei ss ie Pd a ; a) 2 
ree a8 29; Po a ek an 
ov ot ee ‘ 
al hae Eee ert ' 
POT TT a aoe lel a aes eel ‘ ; , y 
eras pret a FI . f eo FI : 
oe Te Oe 1 ds cell s 
ee ee Obert ¥ i Fi ‘. 
ot oe ee SD 4 a a ; a 
i i a S 
wits 2 Lt er , j P 
ae era Re . s, ' F Fy Ls ry - 
, * ¢ 
“ As a 
Se ed 1 a i z 
Ed he he, A A ‘ 
whe, Mmy! PME o RES” rf f 4 iQ be a 
A oh ee al Lad Pate we ite? ae ¥ 
a pe ded ly oe ar '¢ F ° 
Ce aie ied er a be ery ' o* Ps 1 
eal Ti Lal party, 12 ti i ° oq Pr 
rt te Mid aoe tr, Leb rl - r A 
PY thle Seketiee need ky CE Sh she bs ‘ b ‘ 
o te DL ial al Py a TT. ret ra * i id ny ad CY r 
Nes A ia ry . f 
a oh Ae “ P A ‘ 
a Tr Be fe Pte, aA. - Aa a sd F Ls F 
OCR Oe he adh teat aide Ly ! Ps 7) hal pig . cee To : o ¢ “2 
ee re ee TP Pom 7 a regen y ri *. Lh 4 c 
rs Sea ner orion die op Se oeat hi . peer. a “Bhan en te th 1? rr f A Li 
rau eee or) a are he TSMC Ta i Se te perlaer e ay s' . al ae bi i 
PPT net i ie See oD eee ee Pet al unedvin tchygtrS? Py a oe LY ¥ si - 
Ca Of pA Ld ‘ i J 
pple, Cre ri HP Ree DF GA FR a 5 Pad Se 2 C a a re ty eS i] 4 Ay F , ’ . 
be ape La Pe rth oy Por) t « bs 
rarer 7h tae . weed vat net Pera rT ey : F fc 4e@? ud . 
Pere tt Os ied ee TID 1d pak eee arti pp eee - Ba ae Ath bali gl vo ovee 8 A 
rie ett Tos tele she Mt PS TOL NERY Bs hh vel eer’ ee Sr Y ALS SL Se 4 ce ve , . "i ' 
- eT ee TL ok be pee herr Sd Li Se ee TR ee Sey Pia sees, ce Hd varina > 0! Fj re ee: 
Praia he ache Lk ide ad AT ot eal heed ptt pe fies Pr TU rs an cer ee AL hal or > j i” 
a gry PCC PL hee RePrert ry tha te toe Tee atl Eten Ut ee ne ere 7 ee F eb So pabdadd oie aa 8 Ht F J 
fen eek , eT lh ah te ie be mre ee thd Se sash epem aieer® HT) go's te i ae ae he ‘ F 4 ; 
Pha PTT tad eee h tepen Pi Rvcw se sgts we oregisver pigee 9 ftyer owere i < etee Mt palin iyo! b 
& Pa I ated ee hee Ah tie aed Paes : by § 7 * base 7 
; rf esa ee OTE d Macatee de bee Pry Lome Urns Le : fs eas = to F 
; Prr) bs rigs et gterotans a ES le! Sid ha o ae 
et * 7 i J er 
ats F 
Pretty er Irn 2 i hd earns - 8 o 
ty Prue tt ke kl aa re ae @ " 
seein phe peer ty lars 5 Pet a, Pr : ; “ = 
LL ee bd et hd t . e) r RA ii bs LU 
apa li) aM A rr De pee ple Me Sen bed ve i a a 7 
1 et Tobit thks ees at ove meaas Tithit stake Th 4 ’ ] A A 
‘ Pr Lai bY es oe Per) ted ay | Ly 
PPPS ae nL lh wer be 2 ih " oa ‘ 
. tg farneryy yace reel a - abet . it : 
rer emer tk Mas thee Ne! PRY ta tat) hs AL Ae 2 CO) 2 f : 
L Le Gh pie es Tet i haa < 5 P P py 
e preys rr 2 ies % , * 6 Py ° 1 
rreta'abornle Yt 189 
“ ¥ L a iu 
urge te cutee e - 
\ altel f oh Tab > r R 
e  Abegelipted r PE eS ie al aes , ris 4 
PTT) ea tad beets abe Pe SOP if iM ou ie 4 é ati fe é et ‘8 ry b . 
Er te heen eal OLDE teed ee ra wiseneds! J ae ee r A i 
peererer. tt ir Prverre ys It) te dite Di het ele hed . ar ee FE L 
gourarecomeny ip? we erry e yaa seiysonwaatenemncecy sete OTE Co Ne a F i ’ ea 
1S Mies CP aa | Pree kr vi Chega le erh - \- a ‘ ry . 7 z 
Gt Seas by Saree aha evines ogee ats a2) rey a) Ft A ie, ; - i ae 7 
Se ee be ihre hs . Ped eee . A 
y bh: se Nat eT Ay ‘osee tee 6. 9° de wera le one A Ht Ate : dy 8 Ps La ae "i 
sy bh eae Yi ated beg te ; begiv.e settle | 978 8 F 5 - 
tle pper thes My Yar tae eae al cd Hie be hn iar PTY hd ca eo. , 5 D 
nd bed rf Piette Wi ne rare! We tet 1 Ne eh uae Mace ere cea Pyrat oe tS feyg vt  oeig't Y ; 
tT th Peter vetwivtye ° Py Mer Ld pied i SA 7 f oe ig 
ry Pere ek OT i. | ed, el ru@e r L 4 hy te wrislate™= F 
Reve te te Udi debe pee eT [a = Peete SY 2 oe hd prictetet year Phos B19! er RO ‘if | : , s 
ere PAT me Sd Leela Py se) Hier mt babes ot or Pre ary Ll Ak ke a Sd re pai Sr a es : oo P 
RS i ee ad hhh died PP Lh Lh deh aler He Me STITT te MT tt hh Farrar TY 7 1h Lad 3s We tbo a ‘pare ae Z D ' ° 
rts - +4 ee uy opts biergFe grere! rj = ip i! s LU 
esd ede Shhh Ad hy we wn er 5 rs : 
@ropyre tee or at me ny ae ein seit Heer : ane DOC ai , heel mh i ai " y : 
rrr Thee, : weLy tees OTF Mle ” @ 
beeen TAL ered Ti a ad VP Ieee a eee ri e Pf L , LS 
a “4 "9 ey J gs | Fl 
sreteceamiarietr ee td es tieh Re re See: 
afta Nae Tee a eT a Re cone 9 se 
de meg a® miele sea nperiperwer Veer! me Pris iat oe etl hd Uda t e ae 4 Je r S ; ’ c Ae ah ; 
peor ee Ltd hod th Be are Ta hee Pee Paras pA nt t | et fot, te of ies 7, a si * r ; 
Tap DRA Et eh ae Le Phd LL eal See Te a ete Sor TA Drie eas ar he ee aa ame H s ke 7] hl 
bf aed fob trey es meee te PET List 44 4s leh rir irs toa grese he © ¥ mas Pee nF 
epee err te ae = Pdi“ 4 lad ted dt poy re ait ot Ro p> <a 4 Ab f ve ep G7 5 a ij 
Ses agar ste AA aH TR is CR . Ws tga Wr 
hed elo tr bbc hs aa eee oo Ae Pra Py ede hee ane eh ab LS Oo ea LM i 
Phat erat re iit ett pert qin OS Cala Sa dad pk rived Lb 5 ie Pata. et ved A i ¥% os Y a ae r OM ; r 
ee v eet ei ucille dee i) ere JN Ap el: Py See PA erm te > eh tape oo oS 
Ate : BAP Tl Thi ld etal re ere P Mroks b Selah ss LORY Pa oe eh Saleh tr a Pat eae Sct Be 
rR aa) eK tary Tee teat t) hr hey Pea al ee an ong ‘ 
LS he otha 4 iter ie BT ol pe Abed A Leo PT aad a a 7 y 4 2 |, : i 
; OL Th dakd. Matt did ent ol ie ahs LU , j 
viet Herp bata teeeh eets ty rr) Tere Pus mh) ae YY varie 1 4 A a i é 
et tiem oe tay RICA tea He a l J 
oge waatey! BLS tafe PAL stk nat te e 20% i ’ a 
ia Dae] vi ee ae wd # OT es Ca “ re , 

Bead | o Pr hee aoe i ir "1 a 
oT tT Pablo as yverriiey Me tate Te TD eh Bs oti oe Aihg - 
wid Sertoacwe OOF ere test) # ped b 

Pree ett hae a teak ET Lalla Ch tet Lb iy oe g 
raat nytt py one 04) dhe bt 4 HEE tie ich a ed ie ; - 
Te Th ad 04th ea Ade (ed Ee CMEC ULE rerre . ai r 
meres ta Hb pei ¥ topes <A 1d CYTO LAGU Cait owe! ' 
PrP ls oh pene Cel). trad De ARES LU we) dete) by | ’ ‘ 5 
OY orig th kel Uk be Wt PW Ti es a te geld bash Shee hL Py hit Ce ok ty t 
igs p A * 
Tree ee Lhe pk jete: gare Se See a PL 7 a ‘ ri 
fevnei nse: weuteerR el eset y™ | Lint (thie Wi Sod Behe Sh ah At aD t , 
eh eA ed od eeu? he Th he dads te ot dad ve cure" gs & re i i An : 
a eT ar len hte leh Fi joe abe A eT e ® } : iH 
Ta Sigld Lped sam Aaae fie MT te tsi ll ati at hed x a A ind Pies th Se p i ars f ( F 
cpt eben tot T IT shiv a at dt Ba ae iti ae AL - ea . 
ve eect Aa te thei ed SOR YNTETO Fs. tb ag ate . - 
7 nenenee at we ‘a 7 P Md i, Bs ‘ f 1 @ 
Gt errata Piet eee et: ee sy i 
trent ie cd tila cs ee Lhd -ay yD py Ab Wee a ko shalt = be , i‘ 2 at 
aot dh bs EL Lote PEs tld a ep Fal Pa fou : : 
eer heal a talt , RLS fe ; et cag , F ; 
Py] re eas TTT a ache ro race el : 
viareg: @ erinseve ive. 76 ee eee Tt Sy Td CU LAY > ‘ 
s DOTS Ered rd le ct SPALL ae Ab Ns P : : : t 
68h my § and As Vor ON | , re 
aR OE aks eres ake Ba . vert ete ef % oe b ry 
ae, Rte f bp rh te a ee : j : 
at Pech teu ereetniiv ‘Ore Ea ae n 
ww Tte}t r e : H 
 » eS 
if r 
Pod | : : 





on 


